It's Numbers All the Way Through: From Prompt to Token

Your prompt starts as text and ends as text, but in between it is nothing but numbers. The tokenizer turns words into integers; a lookup turns integers into vectors; and from there until the very last step the model is pure linear algebra — matrices multiplying vectors, deterministically. There are exactly two places where symbols and numbers trade sides, and exactly one place where anything random happens.

Sharpening the mental model

A common sketch of this pipeline is almost right, and the almost is where the intuition matters. Three corrections make the rest click.

The tokenizer outputs integers, not vectors. Token IDs are indices into a vocabulary. The vectors arrive one step later, from an embedding lookup that maps each ID to a row of a matrix.
Tokenization isn't the probabilistic/iterative part. Tokenizing the prompt is deterministic and one-shot (BPE greedy merges). The probabilistic, iterative part is the output being grown token by token — the decode loop.
The forward pass is deterministic. It's matmuls plus fixed nonlinearities (GELU, LayerNorm, attention softmax). Probability enters only at the end — output softmax, then one random sample.

Two boundaries, one random draw

The full path. Grey boxes are discrete (text, integer IDs, the emitted token); teal boxes are continuous numerical tensors. Symbols become numbers at embed and numbers become a symbol at sample — and only sample (the orange step) is random. Everything inside the dashed zone is deterministic linear algebra. Then the chosen token is appended and the whole thing runs again — that loop, not the tokenizer, is the iteration.

discrete symbols continuous / numerical the one random step

Inside the tokenizer

"Tokenize" is itself a little pipeline. It never looks at meaning — it's string surgery driven by a fixed, learned merge table, and it is completely deterministic. Here is the whole of it for "Hello, world".

The tokenizer is five string operations. A regex splits text into chunks (keeping each leading space attached, shown as Ġ); byte-level encoding makes any input representable (emoji, code, other scripts — nothing is ever out-of-vocabulary); then BPE merges glue byte pairs back into common subwords; finally each subword is looked up to its integer ID. Step 4 is the only interesting one — expanded below.

Step 4, in slow motion: the merge loop

BPE starts every chunk as its individual characters, then repeatedly glues the highest-priority adjacent pair found in a learned merge table, over and over, until no adjacent pair is in the table. Lower rank = learned earlier = applied first. Watch a chunk collapse from five pieces to one:

Byte-pair encoding is a greedy merge loop. Start from characters, and at each step glue the adjacent pair with the best (lowest) rank in the learned table — here er, then ow, then low, then lower — stopping when no adjacent pair is mergeable. Deterministic, no probabilities, no meaning: just table lookups. A different vocabulary's table would cut the same word differently.

Why words and tokens don't line up

The vocabulary is a fixed set (typically ~50k–200k entries), so the merges only reach as far as the table was trained to. Common strings collapse to a single token; rare or novel ones stop partway and stay split — down to individual bytes in the worst case, which is why nothing is ever "unknown":

Text	Tokens (illustrative)	Count
`" the"`	`[" the"]`	1
`" tokenization"`	`[" token", "ization"]`	2
`" antidisestablishmentarianism"`	`[" anti","dis","establish","ment","arian","ism"]`	6
`"🦁"`	bytes → 3–4 tokens	3–4

This is why token counts (and cost) don't equal word counts, why models have odd blind spots on spelling and arithmetic, and why the very same sentence can be more or fewer tokens in a different model's tokenizer.

Same string, two vocabularies, different cuts

Because the merge table is learned from a corpus, a different tokenizer cuts the identical string in different places and lands on a different token count. Here is one word, run through a small older vocabulary and a large newer one — same twelve letters, different scissors:

Different table, different scissors. Neither is "more correct" — each just applies the merges it happens to have. A larger vocabulary tends to swallow more of a word in one token (fewer tokens, but a bigger embedding matrix to store); a smaller one falls back to shorter pieces. The text never changed; only the ruler did.

The gap widens fast for anything off the English-common path — accents, other scripts, and emoji are where an English-centric vocabulary shatters into raw bytes while a multilingual one stays compact (illustrative counts):

String	A · ~50k, English-centric	B · ~200k, multilingual
`"unbelievable"`	un · bel · iev · able (4)	unbelievable (1)
`"naïveté"`	na · ï · ve · té → bytes (~6)	na · ïveté (~2)
`"東京" (Tokyo)`	bytes (~6)	東京 (~2)
`"🦁"`	bytes (3–4)	🦁 (1–2)

Same characters, different rulers — which is exactly why you can't compare two models' "context length" or "price per token" without also knowing their tokenizers, and why non-English users often pay more tokens for the same sentence.

What each step actually does

#	Step	Transform	What happens
1	Tokenize	text → int[]	Deterministic BPE splits the string into subword units and maps each to its vocab index. No vectors, no randomness.
2	Embed	int[] → vectors	Each ID selects a row of the embedding matrix `E (\\|V\\|×d)`. That row is the token's vector. From here on it is all numbers.
3	Transformer ×L	vectors → vectors	Matmuls (QKV, attention `softmax(QKᵀ/√d)·V`, output projection, two FFN matmuls) + fixed nonlinearities. Deterministic.
4	Logits + softmax	vector → distribution	A linear head projects to one score per vocab word (logits); softmax makes a probability distribution. Deterministic.
5	Sample	distribution → one int	The one random draw: pick a token (temperature / top-k / top-p). Argmax instead = deterministic.
6	Append & repeat	loop	The new token is appended and the model runs again. This is the iteration — the output, not the tokenizer.

The two conversions, up close

Boundary 1 — embed. The integer isn't fed to the network; it addresses the embedding table. Row 995 is the vector for token 995. This lookup is the moment a discrete symbol becomes a point in continuous space.

Boundary 2 — sample. Softmax is deterministic: it turns logits into a distribution. The draw from that distribution (orange) is the only random act in the whole pipeline — and it's what converts numbers back into a token. Choose the tallest bar every time (argmax / greedy) and even this becomes deterministic.

Myth vs. mechanism

Idea	Common sketch	What actually happens
Tokenizer output	a sequence of vectors	a sequence of integer IDs (vectors come from the embedding lookup)
Tokenization	probabilistic / iterative	deterministic, single pass (BPE greedy merges)
Where probability lives	throughout the model	only the output: softmax makes a distribution, sampling draws from it
The "iteration"	growing the tokens	the autoregressive decode loop — one forward pass per output token

Why this framing pays off

Once you see that everything between the two boundaries is tensors, the entire inference-systems story falls into place: it runs on GPUs because it is matrix multiplication; it is memory-bound in decode because each token re-reads the weight matrices; and every acceleration — batching, quantization, low-rank, KV caching — is an operation on those same matrices and vectors. The tokenizer and the sampler are the thin discrete shells; the engine in between is linear algebra.

The companion piece, The Tokenizer Tax, goes deeper on the tokenizer itself — its bottlenecks, how to speed it up, and how it is baked into the training loss. If your team is sizing or debugging an inference stack, that's the work I do.