Sankur Kundu

switchyard · mixture-of-experts serving

64 experts. 2 GPUs.

Designed, built and measured by Sankur KunduOpen source (Apache-2.0) · page updated 15 Oct 2026

OLMoE-1B-7B sends every token to 8 of the 64 experts in each layer. switchyard gives each GPU half of every layer's experts and moves tokens between GPUs with its own kernels, so the output matches one GPU bit for bit. Everything below the model is written from scratch, including the measurement of what load balancing is worth.

tokensrouter · top-8 GPU 0 · experts 0–31GPU 1 · experts 32–63combine peer-to-peer, no NCCL
2.2×Faster decoding than Hugging Face transformers on the same two T4s: 112 vs 52 tokens/s at batch 8
Bit-identicalExpert-parallel output equals the single-GPU layer exactly
5.4–7×Faster decode MoE layer than a per-expert PyTorch loop; 1.9–2.3× a cuBLAS loop
73%Of the T4's memory bandwidth reached by the grouped GEMV that streams expert weights
01 · Serving

Twice as fast as the reference

OLMoE-1B-7B on two Tesla T4s, decode tokens per second at batch 8; 823 tokens/s at batch 64. Peak memory 7.6 GiB per GPU for a 13.8 GB model.

Higher is better (bench/serve.py). Greedy output identical to transformers on 7 of 8 prompts over 32 tokens.

02 · Kernels and dispatch

No atomics, no NCCL

Each GPU is driven by its own thread; getting that right meant finding two cross-stream races that only appeared without CUDA_LAUNCH_BLOCKING.

Route and sort

Each token routed in one warp; rows sorted by expert with a block-wide prefix scan and no atomics.

One launch for all experts

A tensor-core grouped GEMM with SwiGLU fused into its epilogue; for decode, a grouped GEMV at 73% of memory bandwidth.

Peer-to-peer dispatch

A kernel writes rows straight into the other GPU's memory, ordered by CUDA events: 1.25× NCCL's all-to-all at 256 tokens per GPU, level at 16–64, slower at 1 and 1,024.

03 · Load balance

What dropping tokens costs

WikiText-2, 2,048 tokens per GPU. The busiest expert gets 4.0× the mean load; the GPUs stay within 6%. Capacity limits drop the tokens an expert has no room for:

Capacity factorTokens droppedPerplexity
dropless (switchyard)0%12.85
2.09.6%20.19
1.028.6%92.14

Even a generous capacity factor of 2 raises perplexity from 12.9 to 20.2, so switchyard never drops a token (bench/balance.py).

What switchyard adds

Expert parallelism from the kernels up

How mixture-of-experts models are served at scale, rebuilt on two free GPUs, with the costs measured rather than assumed.

  1. Grouped expert kernels

    All 64 experts in one launch, SwiGLU fused, 5.4–7× a per-expert loop.

  2. Exact expert parallelism

    Peer-to-peer dispatch whose output is bit-identical to one GPU.

  3. 2.2× Hugging Face

    112 vs 52 tokens/s decoding OLMoE-1B-7B on two T4s.

  4. Load balance, measured

    Capacity factor 2 drops 9.6% of tokens and raises perplexity 12.9 → 20.2.

Where it falls short

Over PCIe, splitting the experts is never faster than keeping all 64 on one GPU; here expert parallelism buys memory (7.6 GiB per GPU for a 13.8 GB model), not speed.

Open sourceswitchyard on GitHubCUDA kernels, dispatch, benchmarks and raw results.RelatedrelayDisaggregated LLM serving in C++, CUDA and Swift.CiteSoftware citationApache-2.0; reproduce every number from the repository.
@software{kundu2026switchyard,
  author  = {Kundu, Sankur},
  title   = {switchyard: mixture-of-experts serving across GPUs},
  year    = {2026},
  url     = {https://github.com/Shakhtar-Sankur/switchyard},
  license = {Apache-2.0}
}
Sankur Kundu

Sankur Kundu

ML systems engineer & independent researcher · Bhubaneswar, India · sankur.kundu.tw@gmail.com

Builds the systems that train and serve language models. Also the author of lipi, a tokenizer that needs fewer tokens than GPT-4o's and Gemini's in 23 of 24 Indian languages; Babelguard, an open early-warning system for scams across languages; ratchet, RL post-training from scratch; kessler, an independent space-traffic audit that runs every day. Fixes merged into Apple's MLX, Meta's ExecuTorch, Hugging Face's OpenEnv and Oxford's PyZX, and pull requests in review at Google's JAX, NVIDIA, MIT, Carnegie Mellon and ETH Zurich.

PortfolioGitHubLinkedIn