A running set of notes on why performance engineering — the discipline of knowing, not just doing — is becoming more valuable as the cost of producing code collapses toward zero. This page is added to over time; entries run in chronological order, earliest first.

Hire for the durable layer, not the perishable one

2026-06-25

Job descriptions in this field increasingly read as a team's worth of tools demanded of a single person: Triton and CUTLASS and Flash Attention and framework internals and torch.compile and XLA and the collective libraries and the low-precision formats and perf modeling and orchestration. These are catch-alls — written because the field moves faster than roles can be defined, and because the hiring team is itself unsure which tools will matter in three months.

The useful distinction is between two layers:

  • The durable layer — roofline reasoning; occupancy, bandwidth, and register-pressure intuition; collective tuning at scale; the profile → tune → re-measure loop. Expensive to build, slow to teach, and it transfers across every new architecture.
  • The perishable layer — the named libraries and APIs. Real, but a matter of weeks to ramp on, and half of it will be renamed or replaced by the next hardware generation.

A catch-all checklist optimizes the wrong variable. It screens out the strongest generalists and signals that the team does not yet know what it needs. The better move — as a candidate and as someone hiring — is to write the role around the problem and the fundamentals, and to probe those fundamentals directly rather than box-checking a tool inventory. What is unteachable in a quarter is the durable layer; the tool list is cheap to acquire on the job.

This is the same principle, from the other direction, as the note that follows: when the cost of doing falls, what remains scarce is the durable, transferable judgment — so that is what is worth hiring for, and worth building a career on.

When doing gets cheap, knowing gets expensive

2026-06-27

Agents have collapsed the cost of doing — producing code, kernels, whole systems — toward zero. What they cannot collapse is the cost of knowing: what to build, what "right" looks like, when a plausible result is physically impossible, and where this exact thing has failed before. The agent does; the engineer knows.

This is not knowing instead of doing. The doing now serves the knowing. You still profile, write, and run experiments — but to ground-truth the machine. Doing is the instrument; knowing is the product. The leverage skill has shifted from writing code to judging and verifying it.

Performance engineering is the sharpest version of this for one reason: performance engineers deal in measured ground truth, not plausible text. Roofline bounds, occupancy, bandwidth counters — these are physical referees that do not hallucinate. An agent will pass every test while leaving a large speedup on the table, because performance is a correctness criterion it cannot self-certify. The perf engineer is, by definition, the human who checks the machine against reality.

Scientific training is the hallucination detector that makes this work: the instinct for "too good to be true," the reflex to sanity-check a result against conservation laws and physical limits. That instinct is exactly what an architecture exploration or a cross-architecture correctness study is built on — measuring, then refusing to believe a number until it agrees with what the hardware can actually do.

Structure is knowledge: geometric networks, Lie symmetries, symplectic maps

2026-06-27

The clearest place to watch knowing beat raw doing is in models built to respect the structure of a problem instead of inferring it from data. It helps to be precise about what a generic neural network actually does: it does not recognize an underlying concept — it categorizes statistically repeated patterns in its training data. Those patterns may be the imprint of a real underlying structure — a symmetry, a conservation law — but they need not be, and nothing in the training guarantees the correspondence. So the network only approximates that structure, only over the region its data covered, and it drifts the moment you step outside that range: it learned the pattern, not the principle. A model with the structure built in skips the categorizing entirely — it holds the principle exactly, everywhere, by construction.

Three threads of this are converging:

  • Geometric neural networks — architectures whose layers are equivariant to the symmetries of the domain (rotation, translation, permutation). The symmetry is a hard constraint of the architecture, not a hoped-for outcome of training.
  • Lie symmetries — the continuous symmetry groups of a physical system, and, through Noether's theorem, the conservation laws they imply. Encoding the Lie group is encoding the conservation law directly into the model.
  • Symplectic maps — structure-preserving integrators and Hamiltonian networks that conserve the symplectic form, so a learned dynamical system stays stable and energy-consistent over long horizons instead of spiraling off the way an unconstrained integrator does.

All three are the same move: put the known physics into the architecture so the optimizer doesn't have to find it by brute force. That is knowing doing the heavy lifting — fewer parameters, less data, and results that hold where a data-fit model would never generalize. And like every theme on this page, these structures are physically checkable: a conserved invariant or a preserved symplectic form is a referee you can measure against, exactly the kind of ground-truth that catches a plausible-but-wrong answer. It is the same discipline behind the plasma-PINN campaign, where the win comes from respecting the operator structure of the physics rather than fitting it blind.

This is also a guardrail against a certain class of hallucination. Because the encoded structure constrains the output space, the physically-impossible answer is not caught after the fact — it is made unrepresentable, or heavily penalized, up front: a symplectic integrator cannot return a trajectory that gains energy from nowhere, an equivariant network cannot break the symmetry it is built on, a PINN's residual penalizes any solution that violates the governing equation. The referee moves one step earlier — from detecting a bad answer to precluding it. The honest limit lives in the word certain: structure only forecloses the errors that violate the structure you actually encoded. An output that respects every constraint and is still wrong — because the data never covered that regime, or the wrong degrees of freedom were left free — passes straight through. Structure shrinks the space of possible errors; it does not eliminate it. It turns "plausible but physically impossible" from a common failure into an impossible one, and leaves "plausible, physical, and still wrong" for measurement to catch.

This is where the new computing helps. Structure-preserving models trade dense general-purpose arithmetic for sparse, irregular, symmetry-structured operators — exactly the kind of workload that maps badly onto today's one-size-fits-all accelerators and well onto purpose-built ones. Reasoning about which engine should run which operator, and co-designing that hardware before it exists, is precisely the pre-silicon, heterogeneous-SoC problem. The math says what structure to preserve; performance engineering says where it can actually run fast — and the two have to be designed together.

Open by default: disclose the idea, keep the judgment

2026-07-02

There is a real tension in doing computing work for a living: the more openly you disclose your ideas and methods, the faster the field moves — yet the instinct is to hoard them as IP, because that is where the value seems to sit. That instinct is mostly wrong, and it is worth saying why.

The strongest reason to disclose is the one this page keeps returning to: rapid community disclosure is one of the best defenses against getting things wrong. A method that is published, criticized, and reproduced by other teams becomes shared ground truth — more eyes find the errors than any one group has, and the good structure (a verified benchmark, a conservation law, a correctness gate) propagates instead of being reinvented, badly, in private. It is the structure-is-knowledge point made earlier on this page, at the scale of a whole field: a commons of verified constraints catches plausible-but-wrong results that a private stash never would. Open disclosure is hallucination mitigation for a community, not just for one model.

The second reason compounds the first: an idea's value multiplies with the number of teams that use it. A cleaner profiling methodology, a portable performance harness, a correctness stress-test — kept private, it helps one project; disclosed, it helps every team working on the problems that actually matter. Faster and more trustworthy computation enables faster and more trustworthy science, and the hard problems worth accelerating — fusion energy reaching net gain, a closed-loop treatment for diabetes — do not care who owns the idea. They progress fastest when the best methods are in the most hands. Hoarding a good idea to capture a sliver of value is a poor trade against the discovery it could have unblocked.

So where does the value go, if you give the ideas away? Into the durable layer. The judgment to know which idea fits this problem; the measured ground truth from your own hardware; the reputation earned by disclosing work that holds up under scrutiny; the execution a paper or a repo never fully captures. Open source settled this long ago — the code is free, and the value lives in the people who can wield it, the trust they have earned, and the problems they can actually close. Disclosing a methodology does not give away the ability to run it on a client's stack at their accuracy bar; it is the advertisement for that ability. Every investigation on this site ships with its open source and its data for exactly that reason — the PENNANT portability methodology is the whole method, in the open, repo and all.

The balance, then, is not "disclose or protect." It is disclose the idea, keep the judgment. Publish the methods, the benchmarks, the correctness gates, and especially the honest negative results — the ones that mitigate error and move the science. Retain value where it actually lives: in knowing, in ground truth, in trust, in execution. That is the same throughline as the rest of these notes — when doing gets cheap and ideas travel freely, what stays scarce, and worth paying for, is the judgment to use them well.