crucible · sandboxes for RL on code
tests it cannot fool.
Designed, built and measured by Sankur KunduOpen source (Apache-2.0) · page updated 15 Oct 2026
Training a model to write code with reinforcement learning means running its code thousands of times per step, safely. crucible builds a disposable sandbox for every answer from Linux primitives, with no virtual machines, and grades it with tests that never share a process with the model's code.
The grader a model cannot game
Unit-test rewards invite reward hacking, so the asserts run outside the sandbox and only plain values come back from it. tests/test_grader.py tries nine hacks against both graders:
| Hack | crucible | Usual grader |
|---|---|---|
| An always-equal object | fail | pass |
| Exiting before the tests run | fail | pass |
| Forged test output | fail | pass |
| sys.exit at import | fail | pass |
| Hacks that earn the reward | 0 of 9 | 4 of 9 |
During GRPO training every answer was also graded the usual way, to count the rewards a hack would have won.
Faster than a bare Python
A warm process with the task's modules already imported forks each sandbox; a live sandbox can itself be forked mid-episode, its memory shared copy-on-write, so an agent can branch several attempts from one state. Four CPUs:
Time to start a sandbox. 8 branches of a 256 MB sandbox use 508 MB instead of 2,304 MB copied (bench/spawn.py, bench/fork.py).
RL on code, graded safely
Served through OpenEnv's interface, crucible graded every answer in GRPO training on ratchet. MBPP, 499 test tasks, two Tesla T4s:
| Greedy pass@1 | Seed 0 | Seed 1 | Seed 2 | Mean |
|---|---|---|---|---|
| Before training | 34.9% | 34.9% | ||
| After 100 GRPO steps | 38.9% | 38.1% | 38.9% | 38.6% |
| Tasks fixed / broken | 44 / 24 | 48 / 32 | 48 / 28 | |
Paired test over the 499 tasks: p = 0.013. 115 of the 140 fixes were answers that already ran and gave wrong results: the gain is mostly correctness, not formatting (scripts/pool_m3.py).
Fork the episode, not the replay
A branching rollout reaches each leaf through a shared prefix (a workspace of 200 files and an index over a 300,000-entry table, 0.33 s to build) and one more step per level. Every tree was built two ways: fork the snapshot at each node, or start a new sandbox per leaf and replay its path. Four CPUs, one leaf at a time:
| Tree | Leaves | Fork | Replay | Gain |
|---|---|---|---|---|
| 4 × 1 | 4 | 0.42 s | 1.56 s | 3.7× |
| 4 × 2 | 16 | 0.97 s | 5.81 s | 6.0× |
| 8 × 2 | 64 | 2.32 s | 23.1 s | 10.0× |
| 4 × 3 | 64 | 2.21 s | 24.2 s | 10.9× |
Tree is branches × depth. Steps run, fork against replay: 5 / 8, 21 / 48, 73 / 192 and 85 / 256. Every leaf ends in the same state either way, but only because these steps are deterministic. When the prefix records a time or a random id, as real environments do, the 16 forked leaves of a 4 × 2 tree share the one prefix the episode had, while 16 replays rebuild 16 different ones (bench/branch.py). A branch also outlives the sandbox it was forked from, so a tree can close a node whose children still run.
Safe rewards for code-writing models
The infrastructure that RL on code depends on, built from Linux primitives and measured against the attacks a model actually finds.
Sandboxes from Linux primitives
Namespaces, limits, no capabilities and seccomp: 5.3 ms per isolated sandbox, no VMs.
A grader that resists hacking
Tests outside the sandbox: 0 of 9 hacks pass, against 4 of 9.
Branching episodes
Fork a live sandbox in 4.5 ms, copy-on-write: a rollout tree grown from forks needs up to 10.9× less environment time than replaying each path, and every leaf keeps the state the episode had.
A measured training gain
34.9% → 38.6% MBPP pass@1, three seeds, p = 0.013.
A 0.5B model and 100 steps, so the gain is a few points, and each run also broke 24 to 32 tasks it had solved before.
@software{kundu2026crucible,
author = {Kundu, Sankur},
title = {crucible: sandboxed, hack-resistant rewards for reinforcement learning on code},
year = {2026},
url = {https://github.com/Shakhtar-Sankur/crucible},
license = {Apache-2.0}
}Sankur Kundu
ML systems engineer & independent researcher · Bhubaneswar, India · sankur.kundu.tw@gmail.com
Builds the systems that train and serve language models. Also the author of lipi, a tokenizer that needs fewer tokens than GPT-4o's and Gemini's in 23 of 24 Indian languages; Babelguard, an open early-warning system for scams across languages; ratchet, RL post-training from scratch; kessler, an independent space-traffic audit that runs every day. Fixes merged into Apple's MLX, Meta's ExecuTorch, Hugging Face's OpenEnv and Oxford's PyZX, and pull requests in review at Google's JAX, NVIDIA, MIT, Carnegie Mellon and ETH Zurich.