Skip to main content
AI Data

Why Do Some Languages Cost More to Run Through an AI Model?

Lifewood Team6 min read

Short answer. Because the same sentence becomes a different number of tokens depending on the language it is written in, and you are billed per token. Research measuring this found tokenization lengths for the same translated text differing by up to 15 times between languages. Since tokens drive price, latency and how much fits in the context window, a language with a high token count costs more to serve, responds more slowly and gets less room for context.

Most people meet this as a billing anomaly. A product runs the same prompt template in English and in Burmese, and the Burmese version costs several times more per request for no obvious reason. Nothing is misconfigured. The difference was decided at the tokenizer, before the model was ever invoked.

This matters commercially because almost every pricing model for language services is per token, and it matters technically because the context window is also measured in tokens. The same document, translated, can fit comfortably in one language and overflow in another.

Key takeaways

  • The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.
  • These disparities persist even for tokenizers intentionally trained for multilingual support. Adding languages to the training mix does not fix it.
  • The cost lands in three places: the price of commercial language services, processing time and latency, and the amount of content that fits as context.
  • Character-level and byte-level models are not immune, showing over four times the difference in encoding length for some language pairs.
  • The ratio between languages is now generally called a token premium.
  • Controlled work across 97 languages found that vocabulary size and pre-tokenization affect the premium, while similarity between tokenizer training data and test data does not.

Quick comparison

EffectMechanismWho noticesAppears in dashboards?
Higher price per requestPer-token billingFinanceYes, as regional cost variance
Slower responsesMore tokens to process and generateUsers in that languageSometimes, as p95 latency by locale
Less usable contextWindow measured in tokensNobody directlyRarely — shows up as quality loss
Lower training throughputMore tokens per exampleML teamYes, as throughput

Why does the same sentence become more tokens in another language?

A tokenizer is trained on a corpus and learns a fixed vocabulary of subword pieces that compress that corpus efficiently. Text resembling the training corpus breaks into few, long pieces. Text that does not resemble it breaks into many short pieces, sometimes down to individual characters or bytes.

So a tokenizer whose vocabulary was learned mostly from English finds whole English words and common English fragments in its vocabulary. Confronted with a language that has different morphology, a different script, or simply was not well represented, it falls back on much smaller pieces. The text carries the same meaning and the same information, and the model charges more to read it.

The important finding is that this is not simply a matter of how much data a language had. Disparities persisted even for tokenizers that were intentionally trained for multilingual support, which rules out the comforting assumption that a multilingual tokenizer solves the problem by construction.

What does the research actually measure?

The foundational measurement defines parity between languages as the same sentence producing similar token counts, and defines the premium as the ratio of token counts between two languages. Measured across 17 tokenizers, the disparities were large and persistent.

The headline number is the up-to-15-times difference for the same text translated between languages. That is the extreme of the range rather than the typical case, and the typical case is still substantial enough to change unit economics. The same work found that character-level and byte-level models, which are sometimes proposed as the fix, still showed over four times the difference in encoding length for some language pairs.

More recent work takes the question apart under controlled conditions. By training roughly 7,000 comparable monolingual tokenizers across 97 languages while varying the tokenization algorithm, vocabulary size and dataset size, it showed that monolingual tokenizers exhibit a wide range of token premiums even after controlling for dataset size, vocabulary size and data content. The useful negative result is that similarity between the tokenizer's training data and the evaluation data does not drive the premium. Vocabulary size and pre-tokenization do.

Where does the cost actually land?

It lands in three places, and they are not interchangeable. Each one constrains a different design decision.

Price. Per-token billing means a language with a 3x token premium costs roughly 3x as much to serve for the same user-visible output. For a consumer product with per-user margins, that can decide whether a market is viable.

Latency. More tokens means more compute per request and more generated steps, so responses arrive more slowly. High token premiums also reduce throughput during training. A user in a high-premium language gets a slower product.

Context. The window is counted in tokens, so a high-premium language gets less usable context for the same window. In retrieval-augmented systems this is the quietest and most damaging effect: the same retrieval budget admits fewer documents, so answer quality degrades for reasons that never appear in a latency or cost dashboard.

What can you do about it?

You cannot change a hosted model's tokenizer, so the practical options divide into measurement, budgeting, and procurement.

Measure the premium for your own content rather than trusting a published table. Take a representative document set, tokenize it in every language you serve with the specific tokenizer your model uses, and record the ratio against your base language. This takes an afternoon and it turns an invisible cost into a number you can put in a plan.

Then budget in tokens, not characters. Context limits, retrieval budgets and per-user cost caps should be set per locale, because a single global limit expressed in characters or words will silently be a different limit in every language.

For teams training their own models, the controlled work points at two concrete levers: choosing a per-language appropriate vocabulary size, and pre-tokenization strategy, including superword tokenizers that merge across whitespace. Both were found to reduce premium effects substantially.

Finally, treat it as a procurement criterion. If you are evaluating models for a multilingual product, the token premium across your actual language set belongs in the comparison next to benchmark accuracy. Two models with identical quality scores can differ materially in what they cost you to run in the languages you care about, and that difference comes from how each one's vocabulary was built — which is a function of what data went into it.

Frequently asked questions

Not only. The measured driver is how well a language matches the tokenizer's learned vocabulary, which correlates with representation in the training corpus but is not identical to it. Writing system and word length matter, and controlled experiments found vocabulary size and pre-tokenization to be significant factors even between comparable monolingual tokenizers.

They reduce it, they do not solve it. The research is explicit that disparities persisted even for tokenizers that were intentionally trained for multilingual support, so a vendor describing a tokenizer as multilingual has not told you what your premium will be.

They narrow the gap but do not close it. The same study found character-level and byte-level models still exhibiting over four times the difference in encoding length for some language pairs.

Tokenize a parallel corpus — the same content in each language you serve — using the tokenizer belonging to the model you actually use, then divide each language's token count by your base language's. Use your own content rather than a generic sample, because domain vocabulary shifts the result.

Yes, and in the same direction. Training and fine-tuning are billed and bounded by tokens as well, so a high-premium language costs more per example to train on and reduces throughput for the same wall-clock budget.

Sources and further reading

  1. Language Model Tokenizers Introduce Unfairness Between Languages — Petrov, La Malfa, Torr and Bibi: the up-to-15-times disparity, persistence under multilingual training, and the cost, latency and context consequences
  2. Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023 proceedings) — the peer-reviewed version of record
  3. Explaining and Mitigating Crosslingual Tokenizer Inequities — Arnett, Chang, Biderman and Bergen: roughly 7,000 tokenizers across 97 languages, and what does and does not drive token premiums

Have an AI or visibility project in mind?

From AI evaluation and human-in-the-loop review to GEO and AEO strategy, our team can help you deploy with confidence and get found in the AI search era.

Talk to our team