The unit of computation is political
A tokenizer decides what a model gets to treat as a unit. For English-heavy corpora, the usual merge statistics often look reasonable. Across Indian scripts, the same defaults can turn a short word into a long sequence of fragments—and silently make training, inference, and evaluation more expensive.
What we measured
For indicTok, the useful metric was not vocabulary size in isolation. I tracked fertility by language, script coverage, unknown-token behavior, and compression against a shared baseline. The goal was a vocabulary that stayed balanced across 22 languages rather than optimizing the average and hiding the tail.
The systems lesson
Tokenization belongs in the performance budget. Longer sequences increase KV-cache pressure, attention cost, and serving latency. A language-aware tokenizer is therefore not only an NLP artifact; it is infrastructure for fairer and faster inference.
The durable pattern is simple: stratify every metric by language, inspect the failures visually, and treat script coverage as a release gate—not a footnote.