← Cool things

Distributed transformer parallelism

How transformer tensors, model state, and work move across GPUs. Follow the partition, the replicated pieces, and the communication that makes the result whole.

Workload and network settings

These settings apply to every figure and calculator below. The estimates explain scaling; they are not measured benchmarks.

Interactive figures

Read straight through or jump using the contents. Each figure has independent playback. Expand its settings to change model sizes and parallel degrees.

Track what is sharded, what stays replicated, which collective repairs the dependency, and whether that collective fits the physical links.

ParametersActivations GradientsOptimizer CommunicationWaiting / bubble

Keyboard: after focusing a figure, Space plays or pauses, left/right arrows step, and R resets. Option+up/down moves through chapters.

All cost outputs are explanatory estimates. Production choices require framework-specific profiling on the target topology. Sources and implementation routes are in Learning resources.