U-09 · pruebas · July 2026

Tokenization and embeddings: the model's front door

The previous post in this series (What is latent space?, Spanish only for now) took the first step for granted: that a word becomes a vector. Time to open that box, because inside it there are two distinct operations that almost always get told as if they were one.

1. Tokenization: splitting text into chunks and assigning each chunk an integer. It is deterministic, has nothing to learn at inference time, and does not use the neural network at all.
2. Embedding: turning each of those integers into a dense vector. It is a table lookup — a table that was learned during training.

The distinction sounds like a nitpick and it isn’t. Nearly every odd LLM behaviour we tend to blame on the model’s “reasoning” —not being able to count the letters in a word, Spanish costing more than English, long numbers going wrong— happens in step 1, before the network has seen anything whatsoever.

Why we don’t split on words

The intuitive option is to split on spaces: one token, one word. Nobody does it, for three reasons.

The first is vocabulary size. If every word needs its own entry in the table, Spanish alone demands hundreds of thousands of entries just for conjugated verb forms: comer, como, comes, comía, comeríamos, comiéndoselo. Each entry is a row of the embedding matrix, and that matrix also gets multiplied at the end of the model to produce output probabilities. A large vocabulary means an expensive model at both ends.

The second is that there will always be a word that isn’t on the list. Proper nouns, typos, new terms, source code. A closed word-level vocabulary turns all of that into the same <UNK> token, and the information is gone before you start. This is, literally, the problem the foundational BPE-for-NLP paper set out to solve: it is called Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow and Birch, ACL 2016), and its stated goal was open-vocabulary translation.

The third reason is the opposite one: splitting on characters solves both problems —tiny vocabulary, nothing left out— but makes sequences much longer. And since attention costs scale with the square of the length (we covered this in the Attention post), multiplying the number of positions by five multiplies the cost of every attention layer by twenty-five.

The consensus solution sits in the middle: subwords. Frequent words get a single token; rare ones decompose into chunks that have been seen before. Nothing falls outside the vocabulary, and sequences don’t explode.

BPE: the algorithm, and why it is so dumb

The most widespread method is Byte Pair Encoding. Its origin has nothing to do with natural language: it is a compression algorithm Philip Gage published in 1994 in The C Users Journal, which Sennrich et al. repurposed twenty-two years later to chop up words. The whole idea fits in one sentence: find the most frequent adjacent pair of symbols in the corpus, merge it into a new symbol, and repeat N times.

from collections import Counter

def train_bpe(corpus, n_merges):
    # Every word starts split into individual characters
    vocab = {tuple(word): freq for word, freq in Counter(corpus).items()}
    merges = []

    for _ in range(n_merges):
        pairs = Counter()
        for symbols, freq in vocab.items():
            for i in range(len(symbols) - 1):
                pairs[symbols[i], symbols[i + 1]] += freq

        if not pairs:
            break

        best = pairs.most_common(1)[0][0]      # the most frequent pair
        merges.append(best)
        vocab = {apply(symbols, best): freq for symbols, freq in vocab.items()}

    return merges                              # order matters: they are applied in sequence

What matters about this algorithm is what it does not do. It doesn’t know what a root is, or a suffix, or a syllable. It doesn’t know Spanish grammar, or that words exist. It just counts byte pairs. That the result often resembles real morphology —splitting com + ing— is a statistical accident: suffixes are frequent, so they get merged early. When it fails to match morphology, nothing breaks either; the model learns to work with whatever chunks it is handed.

Merges are always applied in the order they were learned, and that is the key to understanding the output. A token cannot exist unless every piece it is built from exists first.

It’s worth noting BPE isn’t the only option. SentencePiece (Kudo and Richardson, 2018) was the first to train directly on raw text, without assuming words come separated by spaces — which in Japanese or Chinese simply isn’t the case. And Kudo’s unigram algorithm attacks the same problem from the opposite end: instead of starting from characters and merging upward, it starts from a huge vocabulary and prunes the units that contribute least. In practice, almost every model you use today is BPE or some variant of it.

Try it: BPE step by step

Below is a toy BPE trained on a tiny corpus of Spanish words (gato — cat, gatos — cats, perro — dog, casa — house…). It has exactly eleven merges, in this order:

ga · to · gato · os · ito · ca · sa · casa · pe · rr · perr

Pick a word and apply merges one at a time. Watch which ones end up as a single token and which get shredded:

Three cases are worth comparing:

• gato ends up as a single token. It was in the corpus, it was frequent, and the merges ga + to → gato were learned.
• gatos ends up as two: gato + s. The os merge exists, but it comes after gato, and by the time its turn arrives the o is already inside another token. Order decides the outcome.
• casita ends up shredded into five tokens, despite being a perfectly ordinary Spanish word. It just wasn’t in the tokenizer’s corpus, and none of the learned merges fit.

That third case is the one that matters. A tokenizer is not neutral: it is trained on a corpus, it treats well what that corpus contained and badly everything else.

From toy to reality: actually measuring it

Intuition so far. Let’s measure. OpenAI’s tokenizers are public through tiktoken, so you can check what happens to Spanish in thirty seconds. I wrote eight sentence pairs —my own translations, mixing technical and everyday language— and counted tokens for each version with two tokenizer generations: cl100k_base (GPT-4’s) and o200k_base (used by later models).

Two readings. First: with GPT-4’s tokenizer, the same content in Spanish costs 45% more than in English. Second, more hopeful: with o200k_base that penalty drops to 19%. Someone decided to spend vocabulary on covering other languages better, and it shows.

This is not a quirk of my sample. The reference work is Language Model Tokenizers Introduce Unfairness Between Languages (Petrov, La Malfa, Torr and Bibi, NeurIPS 2023), which measured 17 tokenizers and found differences of up to 15× in the length of the same text depending on language. Spanish, with a Latin alphabet and decent corpus representation, is among the mildest cases. For some languages the penalty is of another order entirely.

And there is a point worth stating plainly, because it is that paper’s central argument: since price per token and context window size are the same for everyone, this disparity translates directly into some language communities paying more, waiting longer, and fitting less context than others. For the same service.

Try it: real splits

These are real splits, computed with tiktoken and pasted here as-is. Compare each Spanish word against its English equivalent, and compare the two tokenizer generations:

Look at railway and unfortunately: one token each. Their Spanish equivalents, ferrocarril and desafortunadamente, cost four and three. It isn’t that they are rarer words in absolute terms; they were rarer in the corpus the tokenizer was trained on.

Why it can’t count the r’s

With this data in front of you, an internet classic explains itself:

User:   how many r's are in "ferrocarril"?
Model:  Two.

The model isn’t counting badly. It has never seen the letters. It saw four integers — and look where the r’s land when you split it:

' fer' | 'roc' | 'arr' | 'il'
    ↑     ↑      ↑↑

The four r’s are spread across three different tokens, and none of them “contains” the concept of the letter r in any accessible way. Asking an LLM to count characters is like asking you to count the pixels in a photo you are looking at: the information is there in some sense, but not in the format in which you perceive it.

The same thing explains arithmetic. Look at 1234567, split as 123 | 456 | 7: three-digit groups left to right. The addition algorithm we learned at school runs the other way, right to left, carrying as it goes — and with this split the structure of units, tens and hundreds simply doesn’t exist in the input. This isn’t armchair speculation: Singh and Strouse (2024) showed that forcing right-to-left grouping consistently improves arithmetic performance, and models like Llama and PaLM chose to give every digit its own token outright.

And today’s date, 2026-07-28, costs six tokens: 202 | 6 | - | 07 | - | 28. If you are pushing thousands of ISO dates into an agent’s context, there is a cost you probably assumed was free.

BPE is not the only method

BPE dominates, but it is not alone. The other two families you will meet in production do the same job — splitting into subwords — on different criteria:

Two points almost nobody spells out:

SentencePiece is not an algorithm, it is a library. It implements both BPE and Unigram, so seeing “SentencePiece” on a model card tells you nothing about which one it uses. Its real contribution is different: treating the input as a raw Unicode stream, with no pre-splitting on whitespace, encoding the space as ▁. That is what makes it language-agnostic, and it is essential for Japanese, Chinese or Thai, which do not separate words with spaces.

Unigram being probabilistic enables something BPE cannot do. Because one string admits several valid splits with different probabilities, you can sample a different one each training epoch as data augmentation — this is Kudo’s (2018) subword regularization. With classic BPE the split is deterministic and that door is closed.

And with Claude you can’t look

Everything measured here comes from tiktoken because OpenAI publishes its vocabularies. Anthropic does not publish Claude’s: there is no merge file to inspect and no way to tokenize locally. You count by calling the API’s count_tokens endpoint — and estimating with tiktoken doesn’t work, since it belongs to a different provider and undercounts.

The detail worth keeping: the count is model-specific, and it changes between versions. Claude Opus 4.7 introduced a new tokenizer, and Sonnet 5, on adopting it, produces roughly 30% more tokens than Sonnet 4.6 for the same text, at the same price per token. It is the cleanest demonstration of this post’s thesis: splitting is not an implementation detail, it is an economic variable that can shift under your feet without a single line of your code changing.

And outside production there is a research line attacking the root of the problem: tokenizer-free models that work directly on bytes (ByT5) or group bytes into patches dynamically based on next-byte entropy (Byte Latent Transformer). They solve letter-counting and the language bias outright, but they make sequences far longer and attention is quadratic — which is why they have not displaced BPE yet.

From integers to vectors: the embedding table

With the text now a list of integers, the network begins. And the first step is simpler than the name suggests: a table lookup.

import numpy as np

VOCAB = 100_277       # how many distinct tokens the model knows (cl100k_base)
D_MODEL = 4_096       # dimensions of the latent space

# This matrix is a learned parameter: it trains like any other weight.
E = np.random.randn(VOCAB, D_MODEL) * 0.02

ids = [1045, 318, 257, 3797]          # the text, already tokenized
vectors = E[ids]                       # (4, 4096) — that's all

E[ids] is not a metaphor: the embedding layer is literally that indexing operation. It is usually described as a multiplication by a one-hot vector, and mathematically it is, but nobody implements it that way because multiplying by a matrix full of zeros to keep one row is throwing money away.

Three consequences follow:

The table is enormous. With cl100k_base’s 100,277 tokens and 4,096 dimensions, that’s over 410 million parameters in the input alone. In small models the embedding matrix can be a non-trivial fraction of the total — and here is the real design tension in a tokenizer: more vocabulary means shorter sequences but a fatter table. Going from cl100k_base to o200k_base (from ~100k to ~200k tokens) is exactly that bet: double the table so that text —especially non-English text— occupies fewer positions.

Many models reuse the table twice. This is weight tying, proposed by Press and Wolf (EACL 2017): the same matrix that turns integers into vectors is used, transposed, to turn the final hidden state into probabilities over the vocabulary. It saves parameters and, per the original paper, improves perplexity. GPT-2 does it; many current models do too.

These vectors know nothing about context yet. The vector for bank coming out of the table is identical in “the river bank” and “the central bank”. It is the starting point, not the meaning. What turns that generic vector into something that distinguishes the two senses is the stack of attention layers that follows — exactly the step we discussed at the end of the previous post, when we separated static embeddings from contextual representations.

One piece missing: position

As described so far there is a hole. The table is indexed by token, so “the dog bit the postman” and “the postman bit the dog” produce exactly the same set of vectors. And attention has no notion of order either: it looks at every position at once. Without fixing this, a transformer would be a very expensive bag of words.

The fix is to inject position explicitly. The original Attention paper added sinusoidal signals to the embedding; today the standard is RoPE (Su et al., 2021), which instead of adding anything rotates the query and key vectors according to their position. It has an elegant property: when you take the dot product between two positions, the rotations cancel such that the result depends only on the relative distance between them, not on where they sit in absolute terms. That is what lets a model trained on short contexts stretch to long ones with reasonable grace.

h = apply_rope(E[ids])   # and from here on, attention layers

How this was done five years ago

It is worth pausing here, because the word “embedding” meant something quite different not so long ago.

Those static vectors were the whole model, not its first layer: you downloaded them pre-trained and fed them into whatever came next. That is where the example everyone has seen comes from, king - man + woman ≈ queen. It was a property demanded of the embedding on its own, because there was nothing behind it to build one. ELMo (2018) was the hinge: the first to give the same word different vectors depending on the sentence it appeared in.

And here is the real conceptual shift, which is easy to read backwards: the modern embedding is deliberately dumb. It is not that the first layer got worse; it is that the intelligence moved. Ten years ago we asked the embedding to capture meaning by itself. Today we ask it to be a good starting point and nothing more, because disambiguating is the attention stack’s job. The E table above is dumber than a 2013 word2vec, and the model around it is incomparably better.

Careful: two different things are called “embedding”

This is the most common confusion in the topic, and it is worth defusing:

1. An LLM’s input layer — the E[ids] table in this post. One vector per token, internal to the model, never surfaced.
2. Sentence embeddings for semantic search — text-embedding-3, bge-m3. One vector per document, from models trained specifically so that cosine distance between two texts means something.

They share the name and the geometric intuition, but not the purpose or the training. The type-2 ones are what power RAG and semantic search — and they are, literally, what this site’s radar uses to avoid publishing the same story twice when two outlets cover it, which I wrote up in detail in the second logbook entry (Spanish).

What to take away

The tokenizer is infrastructure, not intelligence. It is a deterministic compression algorithm from 1994, trained on a particular corpus, that decides what reaches the model. Nothing happening there is “reasoning”, and a good share of the most-cited LLM limitations live in that step.

Tokens are the system’s unit of economics. Cost, latency and context window are all measured in tokens, and how many tokens your text burns depends on a training corpus you didn’t choose and in which your language was probably under-represented. In my sample: 45% more expensive with GPT-4’s tokenizer, 19% with the next one.

The embedding is just a table lookup. There is no magic there; there is a starting point. All the richness we discussed in the latent space post —context, disambiguation, structure— is built afterwards, layer by layer.

What’s next

The next module in the series builds a latent space from scratch with an autoencoder: a network that learns to compress and reconstruct without anyone telling it what each dimension should represent. It is the most visual way to watch a latent space emerge, and the direct conceptual ancestor of the encoder-decoder architecture in the Attention paper.

To play with

This is the exact script that produces the numbers in the chart above. Thirty seconds and one pip install:

pip install tiktoken

import tiktoken

PAIRS = [
    ("El modelo no entiende las palabras: solo ve una lista de números enteros.",
     "The model does not understand words: it only sees a list of integers."),
    ("La tokenización ocurre antes de que la red neuronal haya visto nada.",
     "Tokenization happens before the neural network has seen anything at all."),
    ("Ayer por la tarde estuve arreglando la bicicleta en el garaje de mi hermano.",
     "Yesterday afternoon I was fixing the bicycle in my brother's garage."),
    ("Cada llamada a la API se factura por tokens, tanto de entrada como de salida.",
     "Every API call is billed by tokens, both input and output."),
    ("Si el presupuesto es limitado, conviene medir antes de optimizar cualquier cosa.",
     "If the budget is limited, it is worth measuring before optimizing anything."),
    ("Los embeddings son simplemente filas de una tabla que se ha aprendido durante el entrenamiento.",
     "Embeddings are simply rows of a table that was learned during training."),
    ("El perro del vecino ladra todas las noches y no me deja dormir tranquilo.",
     "The neighbour's dog barks every night and does not let me sleep properly."),
    ("Esta arquitectura reduce la latencia, pero aumenta el coste de infraestructura.",
     "This architecture reduces latency, but increases infrastructure cost."),
]

for name in ("cl100k_base", "o200k_base"):
    enc = tiktoken.get_encoding(name)
    es = sum(len(enc.encode(a)) for a, _ in PAIRS)
    en = sum(len(enc.encode(b)) for _, b in PAIRS)
    print(f"{name:12}  ES {es:4}   EN {en:4}   ratio {es/en:.2f}x")

# And to see the split of anything at all:
enc = tiktoken.get_encoding("cl100k_base")
for text in [" ferrocarril", " railway", "1234567", "2026-07-28"]:
    ids = enc.encode(text)
    print(f"{text!r:16} → {len(ids)} tok  {[enc.decode([i]) for i in ids]}")

Feed it a paragraph of your own in two languages. Then feed it an identifier from your own code, a UUID, or an ISO date: seeing how many tokens a field you assumed was free actually costs is the fastest way to understand why this matters.

If you want to go one level deeper and build the whole tokenizer, the required reference is Andrej Karpathy’s minbpe: a minimal, readable BPE implementation, with the entire lecture in text form and a guided exercise to reproduce GPT-4’s tokenizer.

Sources

• Sennrich, Haddow and Birch — Neural Machine Translation of Rare Words with Subword Units (ACL 2016). The paper that brought BPE to language processing.
• Gage — A New Algorithm for Data Compression, The C Users Journal, 1994. The original BPE, as a compression algorithm.
• Kudo and Richardson — SentencePiece (EMNLP 2018) and Kudo — Subword Regularization (ACL 2018). The unigram alternative and tokenization without space-based pre-segmentation.
• Radford et al. — Language Models are Unsupervised Multitask Learners (2019). GPT-2 and byte-level BPE: a vocabulary of 50,257 = 256 bytes + 50,000 merges + 1 special token.
• Petrov, La Malfa, Torr and Bibi — Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023). 17 tokenizers, up to 15× difference between languages, and the cost-and-context argument.
• Singh and Strouse — Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs (2024). Number splitting and arithmetic.
• Devlin, Chang, Lee and Toutanova — BERT (NAACL 2019). WordPiece and the ## continuation marker.
• Mikolov, Chen, Corrado and Dean — Efficient Estimation of Word Representations in Vector Space (2013). word2vec and static embeddings.
• Pennington, Socher and Manning — GloVe (EMNLP 2014); Bojanowski, Grave, Joulin and Mikolov — Enriching Word Vectors with Subword Information (TACL 2017). The other two static-vector families.
• Peters et al. — Deep Contextualized Word Representations (NAACL 2018). ELMo: the hinge between static and contextual.
• Xue et al. — ByT5 (TACL 2022) and Pagnoni et al. — Byte Latent Transformer (2024). Tokenizer-free models.
• Anthropic — Token counting. The only way to count Claude tokens, since its tokenizer is not public.
• Press and Wolf — Using the Output Embedding to Improve Language Models (EACL 2017). Weight tying.
• Su, Lu, Pan, Murtadha, Wen and Liu — RoFormer: Enhanced Transformer with Rotary Position Embedding (2021). RoPE.
• EleutherAI — Rotary Embeddings: A Relative Revolution. The most readable explanation of why RoPE works.
• Karpathy — minbpe and the lecture Let’s build the GPT Tokenizer.
• OpenAI — tiktoken. The library behind every original measurement in this post.

← back to the rack