relay · disaggregated LLM serving
prefill → decode.
Designed, built and measured by Sankur KunduOpen source (Apache-2.0) · page updated 15 Oct 2026
LLM serving split in two: prompts on prefill workers, tokens on decode workers, and the KV cache streamed between them while the prompt is still being processed. A C++20 and CUDA engine with attention kernels written for it, and a control plane in Swift. Kill a worker and its requests resume elsewhere with exactly the tokens they would have had.
Three layers, each written from scratch
The engine computes, the transfer layer moves the cache, and the control plane decides where each request goes.
Engine · C++20 + CUDA
Paged KV cache, continuous batching with chunked prefill, preemption and prefix caching. Flash attention on tensor cores (WMMA, online softmax) for prompts; flash-decoding over the paged cache for generation, reading each K/V row once for all the query heads that share it.
Transfer · streamed per layer
Each layer's cache is sent as soon as that layer finishes, over TCP or a shared-memory ring. On a GPU a gather kernel packs it into pinned memory on its own copy stream, so the copy overlaps the next layer's compute. A full decode worker says so and recomputes instead of stalling.
Control plane · Swift
Calls the engine through Swift's C++ interoperability: tokenizer and chat templates, an OpenAI-compatible streaming API, a router that sends shared-prefix prompts to the worker that already caches them, and admission control.
The transfer disappears
46 MB of KV cache after a 2,048-token prompt on a T4: time the decode worker waits for it.
Exposed KV-transfer time after a 2,048-token prompt on a Tesla T4: 99–100% of the transfer is hidden behind compute.
Kill it, it keeps going
Connection, heartbeat and progress watchdogs catch a worker that dies, freezes or stalls; each of its requests resumes on another worker from where it stopped.
384 / 384
Requests identical to a fault-free run while workers were SIGKILLed 60 times at random: SmolLM2, 3 seeds, two cluster shapes. None failed.
Bit-identical
The output is the same whether a request is batched, served from the prefix cache, preempted, or split across prefill and decode workers.
3 real bugs
Found by the chaos tests on the way, each fixed with a test that fails without the fix. A deterministic cluster simulation replays every run from a seed.
Disaggregated serving you can read end to end
The architecture large inference providers use, built small enough to read and tested hard enough to trust.
Layer-streamed KV transfer
The cache moves while the prompt is still being processed, hiding 99–100% of the transfer on a T4.
Attention kernels for the paged cache
Flash attention on tensor cores and flash-decoding: 8.7× faster prefill, 5.4× faster decode steps.
Exact recovery from failures
Requests resume on another worker with exactly the tokens they would have had: 384 of 384 through 60 kills.
A Swift control plane over C++
Tokenizer, chat templates and an OpenAI-compatible API in Swift calling the engine directly, matching Hugging Face exactly.
int8 weights help only a single sequence: from batch 8 up, fp16 on tensor cores is faster. relay has not yet been benchmarked against colocated serving or vLLM on several GPUs; that is the next milestone.
@software{kundu2026relay,
author = {Kundu, Sankur},
title = {relay: disaggregated, fault-tolerant LLM serving},
year = {2026},
url = {https://github.com/Shakhtar-Sankur/relay},
license = {Apache-2.0}
}Sankur Kundu
ML systems engineer & independent researcher · Bhubaneswar, India · sankur.kundu.tw@gmail.com
Builds the systems that train and serve language models. Also the author of lipi, a tokenizer that needs fewer tokens than GPT-4o's and Gemini's in 23 of 24 Indian languages; Babelguard, an open early-warning system for scams across languages; ratchet, RL post-training from scratch; kessler, an independent space-traffic audit that runs every day. Fixes merged into Apple's MLX, Meta's ExecuTorch, Hugging Face's OpenEnv and Oxford's PyZX, and pull requests in review at Google's JAX, NVIDIA, MIT, Carnegie Mellon and ETH Zurich.