Short answer. Because the same sentence becomes a different number of tokens depending on the language it is written in, and you are billed per token. Research measuring this found tokenization lengths for the same translated text differing by up to 15 times between languages. Since tokens drive price, latency and how much fits in the context window, a language with a high token count costs more to serve, responds more slowly and gets less room for context.
Most people meet this as a billing anomaly. A product runs the same prompt template in English and in Burmese, and the Burmese version costs several times more per request for no obvious reason. Nothing is misconfigured. The difference was decided at the tokenizer, before the model was ever invoked.
This matters commercially because almost every pricing model for language services is per token, and it matters technically because the context window is also measured in tokens. The same document, translated, can fit comfortably in one language and overflow in another.
Key takeaways
- The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.
- These disparities persist even for tokenizers intentionally trained for multilingual support. Adding languages to the training mix does not fix it.
- The cost lands in three places: the price of commercial language services, processing time and latency, and the amount of content that fits as context.
- Character-level and byte-level models are not immune, showing over four times the difference in encoding length for some language pairs.
- The ratio between languages is now generally called a token premium.
- Controlled work across 97 languages found that vocabulary size and pre-tokenization affect the premium, while similarity between tokenizer training data and test data does not.
Quick comparison
| Effect | Mechanism | Who notices | Appears in dashboards? |
|---|---|---|---|
| Higher price per request | Per-token billing | Finance | Yes, as regional cost variance |
| Slower responses | More tokens to process and generate | Users in that language | Sometimes, as p95 latency by locale |
| Less usable context | Window measured in tokens | Nobody directly | Rarely — shows up as quality loss |
| Lower training throughput | More tokens per example | ML team | Yes, as throughput |
Why does the same sentence become more tokens in another language?
A tokenizer is trained on a corpus and learns a fixed vocabulary of subword pieces that compress that corpus efficiently. Text resembling the training corpus breaks into few, long pieces. Text that does not resemble it breaks into many short pieces, sometimes down to individual characters or bytes.
So a tokenizer whose vocabulary was learned mostly from English finds whole English words and common English fragments in its vocabulary. Confronted with a language that has different morphology, a different script, or simply was not well represented, it falls back on much smaller pieces. The text carries the same meaning and the same information, and the model charges more to read it.
The important finding is that this is not simply a matter of how much data a language had. Disparities persisted even for tokenizers that were intentionally trained for multilingual support, which rules out the comforting assumption that a multilingual tokenizer solves the problem by construction.
What does the research actually measure?
The foundational measurement defines parity between languages as the same sentence producing similar token counts, and defines the premium as the ratio of token counts between two languages. Measured across 17 tokenizers, the disparities were large and persistent.
The headline number is the up-to-15-times difference for the same text translated between languages. That is the extreme of the range rather than the typical case, and the typical case is still substantial enough to change unit economics. The same work found that character-level and byte-level models, which are sometimes proposed as the fix, still showed over four times the difference in encoding length for some language pairs.
More recent work takes the question apart under controlled conditions. By training roughly 7,000 comparable monolingual tokenizers across 97 languages while varying the tokenization algorithm, vocabulary size and dataset size, it showed that monolingual tokenizers exhibit a wide range of token premiums even after controlling for dataset size, vocabulary size and data content. The useful negative result is that similarity between the tokenizer's training data and the evaluation data does not drive the premium. Vocabulary size and pre-tokenization do.
Where does the cost actually land?
It lands in three places, and they are not interchangeable. Each one constrains a different design decision.
Price. Per-token billing means a language with a 3x token premium costs roughly 3x as much to serve for the same user-visible output. For a consumer product with per-user margins, that can decide whether a market is viable.
Latency. More tokens means more compute per request and more generated steps, so responses arrive more slowly. High token premiums also reduce throughput during training. A user in a high-premium language gets a slower product.
Context. The window is counted in tokens, so a high-premium language gets less usable context for the same window. In retrieval-augmented systems this is the quietest and most damaging effect: the same retrieval budget admits fewer documents, so answer quality degrades for reasons that never appear in a latency or cost dashboard.
What can you do about it?
You cannot change a hosted model's tokenizer, so the practical options divide into measurement, budgeting, and procurement.
Measure the premium for your own content rather than trusting a published table. Take a representative document set, tokenize it in every language you serve with the specific tokenizer your model uses, and record the ratio against your base language. This takes an afternoon and it turns an invisible cost into a number you can put in a plan.
Then budget in tokens, not characters. Context limits, retrieval budgets and per-user cost caps should be set per locale, because a single global limit expressed in characters or words will silently be a different limit in every language.
For teams training their own models, the controlled work points at two concrete levers: choosing a per-language appropriate vocabulary size, and pre-tokenization strategy, including superword tokenizers that merge across whitespace. Both were found to reduce premium effects substantially.
Finally, treat it as a procurement criterion. If you are evaluating models for a multilingual product, the token premium across your actual language set belongs in the comparison next to benchmark accuracy. Two models with identical quality scores can differ materially in what they cost you to run in the languages you care about, and that difference comes from how each one's vocabulary was built — which is a function of what data went into it.