Tokenization looks like plumbing — chop text into pieces, hand off integers. It is really a compression scheme, and that choice quietly decides three things the rest of the model can never fully undo: how much a prompt costs, what units the loss is measured in, and an accuracy ceiling on anything that lives below the token boundary. This is the theory, the bottlenecks, the ways to speed it up — and the part almost no one draws: its connection to the loss.

A tokenizer is a learned compressor

A tokenizer maps a string to a sequence of integers and back. The good ones are lossless — byte-level BPE is a bijection on byte strings, so the tokens reconstruct the prompt exactly. What a tokenizer optimizes is compression: given a fixed vocabulary budget, represent the training corpus in as few tokens as possible. Three algorithm families do this differently:

Family Used by How it builds the vocabulary
BPE GPT, Llama-era Greedy bottom-up merges: glue the most frequent adjacent pair, repeatedly. Fast, deterministic; byte-level handles any input.
WordPiece BERT Like BPE, but merges the pair that most increases corpus likelihood, not raw frequency.
Unigram LM SentencePiece, T5 Top-down: start with a huge vocab and prune to maximize likelihood; pick the most-probable segmentation (Viterbi) at inference.

Whatever the family, the same two knobs govern everything downstream: vocabulary size and fertility (tokens produced per word, or per byte). They trade off against each other — and that tradeoff is the first bottleneck.

vocabulary size → cost sequence length ↓ (attention/doc, cost/token) embed + softmax head ↑ rare tokens undertrained ↑ compute-optimal vocab
The vocabulary tradeoff. A bigger vocab means each document becomes fewer tokens — shorter sequences, cheaper attention, more bytes carried per token. But the embedding table and output softmax both scale with |V|, and the rare tail of a huge vocab is seen so seldom it stays undertrained (the "glitch token" failure mode). There is a compute-optimal vocabulary — and recent scaling work shows it should grow with model size, which most models under-provision.

Where tokenization actually hurts

Bottleneck What it is
Fragmentation (fertility) English-centric vocabs shatter other languages, code, and accents into many tokens — more compute, higher cost, and a smaller usable context window.
Character blindness Counting letters, spelling, digit arithmetic live below the token boundary, which the model sees as an atom. The "how many r's in strawberry" failure — structural, not a bug.
Head cost Embedding + output softmax scale with |V|. A 200k-vocab head is a real slice of compute and memory.
Glitch tokens Tokens in the vocab but barely in training (SolidGoldMagikarp) get near-random embeddings and misbehave.
Boundary effects A trailing space re-tokenizes and skews the next-token distribution (tokenization bias); constrained decoding needs token healing.
Rigidity The vocabulary is frozen at pretraining; new domains or languages can only be fragmented, never learned as units.
what the model sees (tokens / IDs) str aw berry = [495, 707, 15717] token boundary — the loss stops here what's hidden inside (characters, never directly supervised) str aw berry Q: "how many r's?" → can't look inside The characters exist in the bytes, but nothing in training ever rewarded seeing them.
The character wall. Because the loss is defined over tokens, the model is optimized to predict tokens, never the letters packed inside them. The information isn't lost — the tokens are reversible to the exact string — but it's opaque to the objective. That's why character- and digit-level tasks are hard even for otherwise strong models, and why the fixes below often mean changing the units.

Making tokenization faster — and less lossy in what matters

Fix What it does
Fast implementations Rust tokenizers and tiktoken with cached merges + batched parallel encoding; pre-tokenize once and memory-map.
Bigger, balanced vocab Larger multilingual vocabs cut tokens-per-document and the non-English penalty — paid for with a bigger head.
Digit-aware tokenization Single-digit or fixed 3-digit grouping (right-to-left) measurably improves arithmetic.
Vocab transplant / expansion Add tokens for a new language or domain and fine-tune the new rows — no full retrain.
Token healing Re-tokenize across the prompt boundary at generation time so a partial token doesn't bias the next step.
Tokenizer-free models Byte/character models (ByT5, CANINE) and learned dynamic patching (MegaByte, Byte Latent Transformer) drop the fixed vocab — curing fertility bias and character blindness, at the cost of longer sequences that hierarchical patching absorbs.

Tokenization is baked into the loss

A language model is trained by per-token cross-entropy: at each position, penalize the negative log-probability of the correct token. So the tokenizer doesn't just feed the model — it defines the units the loss is measured in.

L_token = -(1/T) · Σ_t  log p(x_t | x_<t)        # T = number of tokens;  perplexity = exp(L_token)

This has a consequence people trip over constantly: per-token loss and perplexity are not comparable across tokenizers. A coarser tokenizer packs more into each token, so each prediction is harder — higher per-token loss — but there are fewer predictions. To compare fairly you normalize by the raw text, converting to bits-per-byte:

BPB = NLL_bits / N_bytes = (L_token / ln2) × (T / N_bytes) = (L_token / ln2) × fertility
same text · area = L_token × T = total bits = same BPB A · many tokens, low per-token loss T = 12 × L = low B · few tokens, high per-token loss T = 6 × L = high Same shaded area → same total bits → same bits-per-byte …but very different per-token perplexity — don't compare it!
Why the units matter. Total description length (the shaded area) is what actually measures the model; per-token loss is that area divided by however many pieces this tokenizer chose to cut. Bits-per-byte (or bits-per-character) divides by the text instead, so it's tokenizer-independent — the only honest way to compare two models with different vocabularies.

Compression is the bridge to accuracy

Language modeling is compression: the total bits to encode a corpus is what tracks a model's capability, and lower bits-per-byte correlates with better downstream accuracy far more reliably than raw per-token perplexity. The tokenizer does the first slice of that compression before the model sees anything — so a good tokenizer front-loads easy structure, and the model spends its capacity on the residual. Push vocab too far, though, and the rare-token tail is undertrained: compression up, accuracy down. There is a genuine optimum.

Prompt fidelity vs. tokens used

Two separate things get called "accuracy of the prompt," and it's worth splitting them:

Question Answer
Does tokenizing lose prompt information? No. Byte-level BPE is reversible — the tokens reconstruct the exact prompt (barring lossy normalization).
Then where's the accuracy cost? (1) the loss never supervises inside tokens → a character/digit ceiling; (2) segmentation shifts the model's string probabilities (tokenization bias).
What do "tokens used" cost? Sequence length = compute, dollars, and context consumed; longer sequences also give errors more room to accumulate.
How to compare models fairly? Bits-per-byte for quality; tokens-per-byte (fertility) for cost. Never per-token perplexity across different tokenizers.

The tokenizer is an exchange rate

It converts characters into the model's currency — tokens — and every rate sets four things at once: the cost of a prompt (fertility → sequence length), the units of the loss (per-token cross-entropy), the fair-comparison metric you are forced into (bits-per-byte, not perplexity), and an accuracy floor on everything below the token boundary. Choose it well and the model starts ahead; choose it badly and no amount of parameters buys the characters back.

This pairs with It's Numbers All the Way Through (how a prompt becomes tokens, vectors, and a result). If your team is choosing or adapting a tokenizer — sizing a vocabulary, fixing multilingual fertility, or debugging character-level failures — that's the work I do.

References

  1. R. Sennrich, B. Haddow, A. Birch. "Neural Machine Translation of Rare Words with Subword Units" (BPE). ACL 2016. arXiv:1508.07909
  2. T. Kudo. "Subword Regularization" (Unigram LM). ACL 2018. arXiv:1804.10959 · SentencePiece: arXiv:1808.06226
  3. Y. Wu et al. "Google's Neural Machine Translation System" (WordPiece). 2016. arXiv:1609.08144
  4. L. Xue et al. "ByT5: Towards a token-free future with pre-trained byte-to-byte models." TACL 2022. arXiv:2105.13626
  5. L. Yu et al. "MegaByte: Predicting Million-byte Sequences with Multiscale Transformers." NeurIPS 2023. arXiv:2305.07185
  6. A. Pagnoni et al. "Byte Latent Transformer: Patches Scale Better Than Tokens." 2024. arXiv:2412.09871
  7. C. Tao et al. "Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies." 2024. arXiv:2407.13623
  8. G. Delétang et al. "Language Modeling Is Compression." ICLR 2024. arXiv:2309.10668
  9. J. Rumbelow, M. Watkins. "SolidGoldMagikarp (plus, prompt generation)." 2023.