Old math, new eruption

Backpropagation dates to the 1980s. Neural networks are, mathematically, a decades-old idea: stack layers of matrix multiplies and nonlinearities, then fit the weights by gradient descent. So why did deep learning erupt only in the last ten years? Nothing in the recipe changed. What changed is that three enablers finally arrived together — and each, underneath, is a systems story.

First, a framing that keeps the rest honest.

LLMs are a subset, not the whole

It's tempting to say "AI = LLMs." Keep the nesting straight:

  • Neural networks are the general, differentiable function approximator — the superset.
  • Inside that: MLPs, CNNs (vision), RNNs/LSTMs (sequences), GNNs, diffusion models, and Transformers.
  • LLMs are one highly specialized instance: Transformers trained self-supervised on enormous text corpora to predict the next token.

The systems point: the substrate is shared. Whether it's a small CNN or a 70B LLM, the inner loop is the same dense linear algebra — matmuls and elementwise ops. That's why the same hardware and the same reasoning apply across the whole family. LLMs just pushed the scale until the systems problems became the whole game.

Why now, #1 — data, without the labeling bottleneck

Supervised learning needs labels, and labels don't scale. The unlock was self-supervision: next-token prediction turns raw text itself into the training signal, so a model can consume trillions of unlabeled tokens. The data ceiling jumped from "what we can afford to label" to "what exists." Then scaling laws showed the payoff is predictable — more data + parameters + compute yields smoothly lower loss — which turned "train a bigger model on more data" from a gamble into an investment thesis.

Why now, #2 — the GPU is shaped like the workload

This is the heart of it. A CPU spends most of its transistors making one thread fast: out-of-order execution, deep branch prediction, big caches. A GPU makes the opposite bet — thousands of simple, in-order arithmetic units, and a plan to keep them fed. That bet wins because neural-network math is embarrassingly parallel: a matmul is millions of independent multiply-adds.

Four properties make it fit:

  • Many simple, in-order cores. No costly machinery to extract parallelism from one instruction stream — the workload already supplies all the parallelism (same op, many data). Transistors go to ALUs, not control logic.
  • Latency hiding by overlap — the key trick. An in-order core would normally stall on every memory load. Instead the GPU oversubscribes each core with many groups of threads and a near-free scheduler: the instant one group waits on memory, another that's ready to compute is swapped in. So memory fetches overlap with arithmetic and the ALUs stay busy. The GPU doesn't make memory faster — it makes the wait free by always having other work. (That's also why it tolerates modest clock speeds: the goal is throughput, not single-thread speed.)
  • High-bandwidth memory. All those ALUs would starve on a normal bus, so GPUs pair with HBM. And — a point I've measured from first principles — high bandwidth only pays off with massive parallelism: one thread can't fill it; thousands can. The GPU brings exactly that parallelism.
  • Tensor cores. Dedicated matrix-multiply units — the workload's single hottest operation, baked into silicon.

The result: the GPU pushes both performance ceilings — peak compute and peak memory bandwidth — far above a CPU, for the one shape of computation deep learning needs.

Why now, #3 — the algorithm became GPU-shaped

The easy-to-forget third leg: the Transformer is GPU-friendly by design. Its predecessors (RNNs/LSTMs) processed a sequence step by step — inherently serial, starving a parallel machine. Attention processes the whole sequence in parallel, turning sequence modeling into the big matmuls a GPU devours. Add robust optimizers, mixed-precision training (fewer bytes, faster tensor cores), and mature software (CUDA, PyTorch), and the algorithms finally fit the hardware that finally fit the data.

The synthesis — a reinforcing loop

Deep learning works now because data, hardware, and algorithms became mutually reinforcing: self-supervision supplies effectively unlimited data; the Transformer turns it into parallel matmuls; the GPU executes those matmuls at enormous throughput by hiding memory latency behind arithmetic; and scaling laws promise that pouring in more of all three keeps paying off.

LLMs are simply the point where that loop was scaled until the systems became the binding constraint — memory bandwidth, interconnect, storage, and the cost of every token. That's the territory my other writing maps: why decode is memory-bandwidth-bound, why the KV cache is the real serving limit, and the glossary of how engines fight it. Once you scale the training loop across many GPUs it becomes a distributed-systems problem in its own right — a separate thread worth reading on neural nets as a distributed system.

The runnable roofline that shows a machine's two ceilings — and where a workload sits between them — is open at codeberg.org/srinathv/llm-masters. If your team is trying to turn that reinforcing loop into trustworthy, efficient computation, that's the work I do.

References

  1. "Attention Is All You Need" (the Transformer): https://arxiv.org/abs/1706.03762
  2. Scaling laws for neural language models: https://arxiv.org/abs/2001.08361
  3. The roofline model (compute vs bandwidth ceilings): https://doi.org/10.1145/1498765.1498785
  4. Companion: memory bandwidth needs parallelism · batching on a T4