tandem · distributed training from scratch
in lockstep.
Designed, built and measured by Sankur KunduOpen source (Apache-2.0) · page updated 15 Oct 2026
The parallelisms behind large-model training, on PyTorch tensors and autograd but with no torch.distributed and no NCCL: its own ring all-reduce, reduce-scatter and all-gather underneath DDP, ZeRO-1, 2 and 3, and GPipe and 1F1B pipelines. On two T4 GPUs every one ends with exactly PyTorch's weights.
Bit-identical to PyTorch
bench/run.py verify on two Tesla T4s: after 20 training steps, every mode ends with the same losses and weights as PyTorch.
| tandem mode | Checked against | Weights |
|---|---|---|
| DDP | torch DDP (NCCL) | equal |
| ZeRO-1 | torch DDP (NCCL) | equal |
| ZeRO-2 | torch DDP (NCCL) | equal |
| ZeRO-3 (FSDP-style) | torch DDP (NCCL) | equal |
| Pipeline, 1F1B | one process | equal |
torch FSDP matches torch DDP too, so ZeRO-3 is equal to both.
Collectives without NCCL
The collectives moved into a PyTorch C++ extension: lock-free single-producer, single-consumer rings in shared memory and a progress thread on the CPU; on GPUs, buffers shared through CUDA IPC, ordered by interprocess events and reduced by a fused CUDA kernel, so the host never waits for a piece.
| All-reduce on a Tesla T4 | Python engine | C++ engine | Speed-up |
|---|---|---|---|
| 64 KB | 1.79 ms | 0.64 ms | 2.8× |
| 1 MB | 1.53 ms | 0.83 ms | 1.9× |
| 4 MB | 1.65 ms | 0.75 ms | 2.2× |
| 64 MB | 10.00 ms | 6.24 ms | 1.6× |
Results bit-identical between the two engines.
Every way to split a model
Each parallelism is built on the same three collectives, in the order Megatron-LM and FSDP use them.
DDP, overlapped
Gradient buckets reduced as backward fills them, so communication hides behind compute.
ZeRO-1, 2 and 3
Sharded optimizer state, gradients and, FSDP-style, parameters gathered just before use and prefetched: 17% less memory per GPU than DDP.
GPipe and 1F1B
Pipeline schedules in Megatron-LM's order; 1F1B uses 32–52% less memory than GPipe on an 86M-parameter GPT.
Large-model training, understood from the wire up
What torch.distributed, NCCL, DeepSpeed and Megatron do, rebuilt so that every byte on the wire is accounted for.
Own collectives
Ring all-reduce, reduce-scatter and all-gather within 2% of NCCL's bandwidth at 64–256 MB.
Every parallelism
DDP, ZeRO-1/2/3 and GPipe/1F1B, bit-identical to PyTorch after 20 steps.
A C++/CUDA engine
Shared-memory rings, CUDA IPC and a fused reduce kernel: 1.6–2.8× faster.
Memory, measured
ZeRO-3 17% less than DDP; 1F1B 32–52% less than GPipe.
tandem's DDP reaches 86% of PyTorch DDP's throughput, and the C++ engine has not yet been timed against NCCL across two GPUs.
@software{kundu2026tandem,
author = {Kundu, Sankur},
title = {tandem: distributed training on its own collectives},
year = {2026},
url = {https://github.com/Shakhtar-Sankur/tandem},
license = {Apache-2.0}
}Sankur Kundu
ML systems engineer & independent researcher · Bhubaneswar, India · sankur.kundu.tw@gmail.com
Builds the systems that train and serve language models. Also the author of lipi, a tokenizer that needs fewer tokens than GPT-4o's and Gemini's in 23 of 24 Indian languages; Babelguard, an open early-warning system for scams across languages; ratchet, RL post-training from scratch; kessler, an independent space-traffic audit that runs every day. Fixes merged into Apple's MLX, Meta's ExecuTorch, Hugging Face's OpenEnv and Oxford's PyZX, and pull requests in review at Google's JAX, NVIDIA, MIT, Carnegie Mellon and ETH Zurich.