lipi · a study of 24 Indian languages
The token tax.
Research & tokenizer by Sankur KunduFirst published 15 October 2026 · paper public 16 October 2026, arXiv (cs.CL) forthcoming
AI models read, write, time and bill text in tokens. Their tokenizers were built mostly on English, so the same sentence costs far more tokens in an Indian language: on GPT-4o, Odia needs 4.99× the tokens of English, and Santali 13.7×. lipi measured that tax across 15 tokenizers, then removed it from an open model.
TAX INVOICE
FLORES-200 devtest · sentence 67 · billed by GPT-4o
“Middle order batsmen, Sachin Tendulkar and Rahul Dravid, performed well and made a hundred-run partnership.”
Re-issued with lipi-Indic 96K
Try any sentence
Pick a language, or type your own. The real tokenizers of GPT-4o, Qwen 2.5 (as it ships) and lipi-Indic split it in your browser; each block is one token. Nothing is sent anywhere.
Loading tokenizers…
Every language, every model
Tokens for the 1,012 FLORES-200 devtest sentences in each language, divided by the tokens the same tokenizer needs for the same sentences in English. 1× means no tax.
Read it as a table
Time, memory and money
A token count is not a price, since every API sets its own. It is how much text fits in a model's memory, and how many steps the model takes to write an answer. Measured on Qwen 2.5 0.5B, one T4 GPU:
Writing an Odia sentence: 8.37 s with Qwen's own tokenizer, 1.19 s with lipi-Indic. English takes 0.83 s.
Odia sentences that fit in a 32K-token context window, before and after.
Reading a 3,000-character Odia document before the first word of the answer: 3.40 s → 0.088 s (5,923 tokens → 650).
Where the tax comes from. Most tokenizers have few tokens for Indian scripts, so words fall apart into pieces or raw bytes. Worse, the pre-tokenizer of GPT-4, Llama 3 and Qwen splits words at every vowel sign before the vocabulary is even consulted: however large their vocabulary, they cannot write Odia in fewer than 2.33× the tokens of English.
The fix. lipi-Indic changes that splitting rule for Indian scripts only, and adds 96,000 tokens learned from text in all 24 languages to Qwen 2.5. Every existing token keeps its id, so English and code are tokenized exactly as before, and with a smaller vocabulary than Gemini's it needs fewer tokens than both GPT-4o's and Gemini's tokenizers in 23 of the 24 languages.
Measured, explained, fixed
lipi focuses on India: every scheduled language, on the tokenizers that ship with the models people actually use. It finds where the tax comes from, and removes it from an open model.
A census of the tax
All 22 scheduled languages of India (24 language–script pairs, 1.16 billion speakers) on 15 tokenizers that ship with today's models, from GPT-4o and Gemini to Llama, DeepSeek, Mistral, Sarvam and Krutrim, on the same 1,012 sentences per language.
The cause, isolated
The tax has two sources: vocabularies with few Indian-script tokens, and a pre-tokenizer floor that splits words at every vowel sign before the vocabulary is consulted. lipi measures that floor for every Indian language (Odia cannot go below 2.33× English on GPT-4, Llama 3 or Qwen, whatever their vocabulary). The floor was formalised first by Regmi, Pudasaini and Phakami Pun; lipi credits them.
A fix that keeps every token id
lipi-Indic changes the split rule for Indian scripts only and adds 96,000 merges to Qwen 2.5. Every existing token keeps its id, so English and code are tokenized exactly as before. With a smaller vocabulary than Gemini's, it needs fewer tokens than GPT-4o's and Gemini's in 23 of 24 languages.
The payoff, measured end to end
On a real model and GPU: Odia written 7.0× faster, 7.6× more Odia text in the same context window, and a long Odia document read 39× sooner.
The story in five facts
Free to quote and reuse with credit to “Sankur Kundu, lipi”. Every figure links back to raw data.
- On GPT-4o, the same sentence takes 4.99× as many tokens in Odia as in English, and 13.7× in Santali.
- Weighted by speakers, an average Indian's text costs 1.92× English on GPT-4o, 2.4× on Llama 4 and 5.3× on Qwen 3.
- On Llama 3 and Qwen 3, every Indian language measured costs at least 2× English.
- Some tokenizers cannot get below 2.33× for Odia however large their vocabulary, because of how they split words.
- An open fix, lipi-Indic, brings Odia to 1.35× and the speaker-weighted average to 1.30×, and writes Odia 7× faster.
Contact
Sankur Kundu, independent researcher, Bhubaneswar, Odisha. Available for interviews and background briefings.
@misc{kundu2026lipi,
author = {Kundu, Sankur},
title = {Removing the Token Tax from an Existing Model: A Vocabulary Extension for 24 Indian Languages That Keeps Every Other Token},
year = {2026},
howpublished = {\url{https://github.com/Shakhtar-Sankur/lipi}},
note = {Tokenizer: https://huggingface.co/Shakhtar-Sankur/lipi-indic-96k}
}
Sankur Kundu
ML systems engineer & independent researcher · Bhubaneswar, India
Builds ML infrastructure from scratch that matches the world's best, and makes it work for the people today's models leave out. Also the author of Babelguard, an open early-warning system for scams across 49 languages, and ratchet, reinforcement-learning post-training from scratch (GSM8K 48.1% → 52.5%). Fixes merged into Apple's MLX, Meta's ExecuTorch, Hugging Face's OpenEnv and Oxford's PyZX, and pull requests in review at Google's JAX, NVIDIA, MIT, Carnegie Mellon and ETH Zurich.