The Tokenizer Is a Border

Every language model draws a map of language before it reads a word. For Arabic, that map was mostly drawn somewhere else.

الملخص

تُقدَّم مُقطِّعات الوحدات الجزئية بوصفها أنابيب محايدة، وهي أقرب إلى اتفاقيات تجارة: تُتفاوَض مرة واحدة، وقت التدريب المسبق، لصالح من ورّد المدونة. تمشي هذه المقالة عبر افتراضات مفردات ضغط الأزواج، ولماذا يخرقها صرف العربية، وماذا يكلف تضخم الرموز الناتج من سياقٍ ومال، وأي الخيارات الهندسية تعيد فتح التفاوض.

Epistemic status: the mechanisms are textbook; the numbers are ranges, not gospel; the argument is mine.

A map before reading

A language model never sees your sentence. It sees a sequence of integers, and the function that turns one into the other is the tokenizer: a fixed vocabulary of character chunks, learned once from a pretraining corpus and frozen forever after. Everything downstream, attention, scaling laws, pricing pages, inherits that first decision.

Because the decision is made before training, it is invisible in the demo and permanent in production. You can finetune weights. You cannot, in any practical sense, finetune the border.

Label 1Tokenizer: the fixed lookup that maps text to integer IDs. Typical modern vocabularies hold 32k to 256k entries.

What BPE assumes

Byte-pair encoding and its relatives (WordPiece, unigram LM) share one instinct: frequency deserves real estate. Merge the most common adjacent pairs, again and again, until the budget of vocabulary slots is spent. Whatever the corpus repeats becomes a single cheap token; whatever it rarely sees is spelled out in fragments.

This is a reasonable instinct and an unreasonable politics, because the corpus was never neutral. When the training text is overwhelmingly English, the merge table becomes a dictionary of English convenience. The word the earns a private token. An Arabic verb earns shrapnel.

Label 2BPE: byte-pair encoding. Start from bytes or characters; repeatedly merge the most frequent adjacent pair into a new symbol.

The shape of an Arabic word

Arabic morphology is not a longer version of English morphology; it is a different geometry. Meaning lives in a consonantal root, usually three letters, and grammar arrives as a pattern poured around it. From ك ت ب, the root of writing, the language derives كَتَبَ he wrote, كُتُب books, كاتِب writer, مَكتَب office, مَكتَبة library. Five words, one root, five patterns.

Then the clitics attach. Conjunctions, prepositions, the definite article, object pronouns: وَسَنَكتُبُها is a single orthographic word meaning and we will write it. A frequency-based merge table, trained mostly on English, meets that word the way a customs officer meets an unfamiliar passport: it takes it apart.

Label 3The root-and-pattern system is why Arabic dictionaries are organized by root, not by first letter. Tools like CAMeL Tools exist largely to recover this structure.

Fertility, measured politely

The polite name for the damage is fertility: the average number of tokens a tokenizer spends per word. For English under an English-centric vocabulary, fertility sits close to one. For Arabic under the same vocabulary, published audits and my own counts land the figure at roughly one and a half to two and a half times the English rate, worse for dialects, worse again for diacritized text.

Fertility is not an aesthetic complaint. Context windows are quoted in tokens, so an Arabic document fits less of itself into the same window. API prices are quoted per token, so the same paragraph costs more to process. Latency scales with sequence length, so the same answer arrives later. The border charges by the syllable.

Label 4Fertility: tokens per word. A fertility of 2.0 means the model spends twice the sequence length for the same text.

The diacritics ledger

Arabic writing routinely omits the short vowels. كتب on the page may be كَتَبَ he wrote, كُتُب books, or كُتِبَ it was written; the reader disambiguates from context without noticing the work. Pipelines usually strip whatever diacritics do appear, treating tashkeel as noise.

I think the ledger runs the other way. Diacritics are compressed meaning, and restoration is a legitimate auxiliary task: teach the model to put the vowels back and you have taught it, cheaply, a great deal of syntax and semantics. Noise is what you call signal you have decided not to pay for.

The smugglers' road

Meanwhile, actual Arabic usage keeps leaving the checkpoint. Dialects diverge from Modern Standard Arabic in vocabulary and syntax, and a model tuned on MSA news degrades smoothly as the input drifts toward the Gulf, the Levant, or the Maghreb. Further out is Arabizi, Arabic written in Latin letters with digits standing in for missing sounds: 3 for ع, 7 for ح, 2 for the glottal stop.

Arabizi is what a language does when the infrastructure will not carry it: it smuggles itself across in someone else's alphabet. Any system that claims to serve Arabic users but chokes on 3ala lahjtna is serving the language's passport photo, not the language.

Label 5The digit conventions arose on keyboards and phones that lacked Arabic support. They are stable enough to normalize with rules, most of the time.

Cashing the metaphor

Metaphors in technical writing should be invoiced, so here is the bill. A border decides who crosses cheaply: the vocabulary decides which languages compress well. A border is drawn in negotiations the crossers did not attend: the merge table is fixed at pretraining, by whoever chose the corpus. A border collects tariffs: fertility is the tariff, paid in context, money, and milliseconds. And borders create smuggling: Arabizi is the informal economy of a language under-served by formal infrastructure.

The tokenizer is where a model's constitution is written, and most of the world was not in the room.

What actually helps

The border is an engineering decision, which is the good news, because engineering decisions can be remade. Arabic-centric vocabularies help: models in the Jais family and SDAIA's ALLaM negotiated their merge tables with Arabic at the table, and their fertility on Arabic text shows it. Morphology-aware segmentation helps: keeping roots and frequent clitics recoverable as units, rather than accidents of frequency. Byte-level and character-hybrid architectures help at the margin by refusing to hold a fixed vocabulary at all, trading tariff for sequence length.

None of these is free. All of them are choices, which is the point.

Label 6Jais (2023) and ALLaM (2024) are Arabic-centric LLM efforts from the Gulf; both publish enough detail to audit their tokenizer decisions. Byte-level models such as ByT5 sidestep vocabularies entirely.

The Border Audit

If you buy, deploy, or regulate models for Arabic-speaking users, you do not need to take anyone's word. Demand five numbers.

Five numbers before the contract is signed

Tokens per word on your MSA text, side by side with English parity.
The same figure on your dialect mix, not the vendor's.
Accuracy delta with and without diacritics on inputs that carry them.
Cost per served answer in your deployment region, not per token in theirs.
Provenance of every Arabic benchmark quoted: written in Arabic, or translated into it.

A vendor who cannot produce these numbers has not measured them. A vendor who will not produce them has.

A Khobar coda

I write this from Al Khobar, on a coast that learned the hard way what it means to export a raw material and import the refined product at a markup. Raw text out, capability in, tariff at the crossing: the shape of the trade is familiar here.

The Gulf already prices energy by the molecule. It should learn to price language by the token, and then, having learned to price it, refuse to overpay. The border is ours to renegotiate; the first step is to stop calling it geography.

Sara Chrouf

Al Khobar · February 2026

Cite this essay

@misc{chrouf2026border,
  author = {Chrouf, Sara},
  title  = {The Tokenizer Is a Border},
  year   = {2026},
  url    = {https://sarachrouf.com/journal/the-tokenizer-is-a-border}
}