ratchet · lab notebook · found, explained, fixed
The sampler waswas not the policy.
Research & engineering by Sankur KunduPage updated 15 Oct 2026 · technical report public 16 October 2026, arXiv forthcoming
In reinforcement-learning post-training, one engine samples the answers and another computes the gradient, and the maths is only right if both hold the same model. I built both sides from scratch, a C++/CUDA inference engine and a PyTorch trainer, so that every disagreement between them could be seen. The weight sync was silently rounding away most of each update.
Results, Qwen2.5-0.5B on two T4 GPUs
An update too small to see
The engine stores weights in fp16. Around a typical weight of 0.02, fp16 values are 1.5×10⁻⁵ apart, and an AdamW step at learning rate 10⁻⁶ moves a weight by about 10⁻⁶. Rounded, the update disappears. Move the slider.
Rounding at sync is the gap
Each source switched on alone, measured on the same 64 fresh rollouts after 1, 3, 10 and 30 GRPO updates: mean change in per-token log-probability (nats, log scale).
read it as a table
At the released weights every source is below 0.002 on average and rounding to bf16 is exactly zero, because the released weights are bf16 values. "bf16 rounding" rounds the trainer's weights to bf16; it is not a bf16 engine. One run, Qwen2.5-0.5B-Instruct, GSM8K.
Rounding at weight sync against fp16 arithmetic, on average; 15–39× against the engine's own kernels, which never put a token outside the ratio bound.
Share of weights an fp16 engine sees change in an update, from the first to the thirtieth. For bf16: 1.7% → 0.7%.
The gap bf16 rounding would leave by update 30, with 15% of tokens outside the ratio bound; fp16 stays at a mean of 0.012–0.030.
Fast enough to matter on small GPUs
The engine, the trainer and the loop between them, measured on two Tesla T4s.
Weight sync: the engine converts the trainer's weights in place on its own GPU, instead of a reload.
Partial rollouts and one-step-ahead training cut the wait for the longest answer.
A sort-free sampler: the engine had been sorting 151,936 entries on the CPU for every token.
Trainer and engine share one weight layout and agree within 9.5×10⁻⁷; runs reproduce bit for bit.
| Mode (100 GRPO steps, all 1,319 GSM8K test problems) | Before | After | Time |
|---|---|---|---|
| Colocated: both GPUs generate, then train | 48.1% | 52.5% | 50.6 min |
| Split: one generates while the other trains | 47.8% | 50.2% | 44.7 min |
One run per mode; the difference between the two gains is not established.
What this work adds
Most RL stacks measure the engine–trainer gap as one number and correct it in the loss. ratchet controls both sides, so it can say where the gap comes from.
An observable RL loop
A C++/CUDA engine that returns every sampled token's log-probability, and a trainer in the engine's own weight layout, logging the gap on every step of every run.
A silent failure, and its fix
Weight sync into an fp16 engine rounds away most of each update. Computing the trainer with the engine's rounded weights, while the optimizer keeps float32 masters, takes the gap from 5.2 to 0.043 nats.
The cause, isolated
Rounding at sync is 9–22× arithmetic precision and 15–39× the kernels; bf16 would be worse and grows worse, consistent with recent findings that fp16 reduces the mismatch.
Systems for small hardware
In-place sync 369× faster than a reload, 1.9× faster steps and a 27× faster sampler, on two T4s; GSM8K 48.1% → 52.5%.
@software{kundu2026ratchet,
author = {Kundu, Sankur},
title = {ratchet: reinforcement-learning post-training from scratch, and the training--inference mismatch},
year = {2026},
url = {https://github.com/Shakhtar-Sankur/ratchet},
license = {Apache-2.0}
}
Sankur Kundu
ML systems engineer & independent researcher · Bhubaneswar, India · sankur.kundu.tw@gmail.com
Builds the systems that train and serve language models. Also the author of lipi, a tokenizer that needs fewer tokens than GPT-4o's and Gemini's in 23 of 24 Indian languages, and Babelguard, an open early-warning system for scams across languages. Fixes merged into Apple's MLX, Meta's ExecuTorch, Hugging Face's OpenEnv and Oxford's PyZX, and pull requests in review at Google's JAX, NVIDIA, MIT, Carnegie Mellon and ETH Zurich.