A study found AI models use up to 17 times more tokens for less common languages. This raises costs and latency for users and startups. The gap persists even among the top 20 internet languages. Script type heavily influences token counts. Latin-script languages fare better than others. Startups in places like Bengaluru face higher costs than English-speaking competitors. Token limits also reduce context capacity for non-English users. Better training data alone does not fix the tokenizer gap. Labs must design tokenizers to handle diverse scripts better. Experts call for published language-by-language cost scorecards.
What changed
Tokenizers were assumed to treat all languages equally in cost and performance.
What it unlocks
Identifying a hidden cost disparity that affects non-English language users and startups.
- 17x token cost gap for some languages
- 1.5 to 5x token cost gap in top 20 languages
- 200 languages analyzed
- 38 dialects in RESPIN dataset
Sources