What happened

Any such system is designed for specific goals and hard limits. For language models the main goals are to serve more requests and to shorten the wait for an answer. The limits are the speed and types of hardware operations, the size and hierarchy of memory, and the bandwidth, latency, and hierarchy of moving data — inside the machine and across the network. The essay points at the Roofline section of Google’s Scaling Book, which is where you see whether you are bound by arithmetic or by feeding the chip.

The three layers split those limits across different tools. The kernel is scalar, vector, and tile instructions; typical environments are CUDA, C/C++, PTX, and Triton. The graph is the tensor work of one model on one GPU: composability and how easy it is to change the schedule matter here. The system layer is how you split memory, compute, and links across many devices for training and for inference. A trick from a lower floor does not replace a decision on a higher one.

At kernel level the telling example is attention. A naive formula drags a large attention-matrix row through global memory. Online softmax rewrites the same formula as a recurrence, so those extra global reads and writes go away. At graph level, convenient frameworks historically lost to specialists: serious inference used ONNX Runtime, TensorRT, and FasterTransformer. PyTorch has since taken a share of inference because CUDA graphs and torch.compile got cheaper, and the models themselves are so heavy that framework overhead is no longer the main bill. Typical graph moves are fusing neighboring kernels, merging ops, quantization, and sparsity: move less data and use less memory for the same math.

At system level almost any parallelism raises throughput, and not all of it cuts latency. Data parallelism splits batches: almost no communication at inference time, light communication in training, and latency does not drop. Pipeline parallelism splits batches and layers: communication stays light, model memory is saved, latency still does not drop. Tensor parallelism splits fully connected layers by rows and columns and attention by heads: latency goes down, and communication gets expensive. The choice follows whatever you lack — memory or a link. Compute volume usually stays constant; the essay calls out exceptions such as fully sharded data parallelism separately. For tying training and inference together it points at VeRL and a join with Megatron-LM and vLLM, and at the NeMo and DeepSpeed docs for the strategy survey.

Why it matters

Without the map it is easy to speed up the wrong floor. Quantization will not fix expensive communication under tensor parallelism, and an extra GPU will not remove latency if the parallelism scheme is not built to reduce it. Before choosing among “fine-tune”, “compress”, and “spread across cards”, name the limit: arithmetic, memory, or the link.

The second shift is that inference no longer lives in a separate world from training. PyTorch in production inference, and shared stacks such as Megatron-LM with vLLM, mean one team can stay on one toolchain instead of two incompatible chains. That does not retire specialist runtimes where every percent of latency is money. It retires the habit of treating “a convenient graph” and “fast inference” as opposites.

In practice

Read someone else’s speedup and ask which of the three layers produced it. Fused kernels are a single-GPU graph story. Splitting attention heads across devices is a system story, and latency and throughput will diverge. Mixing those floors is a common reason a paper’s speedup does not survive contact with your setup.

The parallelism table in the essay is for that choice, not for turning every strategy on at once. Model memory and the price of communication decide more than the fashion of the method name.

1. Name the bottleneck before you pick a technique: compute, memory, or data movement.
2. On one GPU, look first at fusion, quantization, and whether a kernel is hauling extra data, as in the attention example.
3. If you add devices, check whether latency actually falls. Data and pipeline parallelism do not cut it.
4. Do not keep training and inference on incompatible stacks without a reason — see whether your frameworks join the way Megatron-LM and vLLM are described.

Takeaway

The three-story map does not replace a profile on your hardware, but it stops optimization by guesswork. The kernel removes needless trips to memory, the graph stitches model ops, and the system splits work across devices — and only some of those splits make a single answer faster rather than a batch cheaper. For a team shipping a model, it is a useful map before any specific repository is opened.