Tokenization

Why text must be cut up

Language models do not read characters or words. They read tokens: integer IDs drawn from a fixed vocabulary, each mapped to a vector the model learned during training. Before any text can go into a model, a tokenizer has to convert the raw string into a sequence of these IDs, and after generation, it converts IDs back into text.

The vocabulary is fixed when the model is trained and cannot grow later. This creates a genuine design tension. A small vocabulary means every word gets chopped into many pieces, making sequences long and slow to process. A large vocabulary means shorter sequences but a bigger embedding matrix, which costs memory and needs more training data to fill in. Every tokenizer is a compromise on this tradeoff.

Three granularities

The earliest approach split text into whole words. This is intuitive but breaks on the long tail of language. Any word not in the vocabulary becomes an unknown token, and real text is full of rare words: names, typos, new slang, technical terms. A word-level model of English needs hundreds of thousands of entries and still hits unknowns constantly.

Character-level tokenization goes the other way. The vocabulary is tiny, a hundred or so characters covers most text, and there are never unknowns. The cost is sequence length: a paragraph becomes hundreds of tokens, and the model must learn to assemble meaning from individual letters, which wastes capacity on spelling before it can get to semantics.

Subword tokenization is the middle path that won. Common words become single tokens. Rare words split into meaningful pieces. The word "unhappiness" might become "un", "happi", "ness", or similar fragments, each of which the model has seen in other contexts. Nothing is ever truly unknown, because any string can be decomposed down to characters or bytes. Vocabulary stays in the tens of thousands, sequences stay manageable, and rare words still carry usable structure.

Byte pair encoding

The dominant subword algorithm is byte pair encoding, or BPE. It started life as a data compression method published by Philip Gage in 1994. In 2016, Sennrich, Haddow, and Birch adapted it for neural machine translation in a paper on handling rare words with subword units, and it has been the default choice for language models ever since.

The training algorithm is greedy and simple. Start with a base vocabulary of individual characters. Count every adjacent pair across the training corpus. Merge the most frequent pair into a new token. Repeat until the vocabulary reaches the target size. Frequent words quickly collapse into single tokens, while rare words remain as sequences of pieces.

OpenAI's GPT-2, released in 2019, introduced byte-level BPE: the base vocabulary is the 256 byte values rather than characters. This guarantees that any text in any language, including emoji and code, can be tokenized with zero unknown tokens. GPT-2's vocabulary was 50,257 tokens. GPT-4's tokenizer expanded this to about 100,256. BERT uses a related algorithm called WordPiece with a 30,522-token vocabulary. Llama 2 uses SentencePiece with 32,000.

Encoding new text applies the learned merges in the order they were learned. The process is fully deterministic: the same string always produces the same tokens. Decoding just concatenates the pieces back together, so tokenization is lossless.

What this means in practice

A rough rule of thumb for English is that one token covers about four characters, or around three quarters of a word. "Tokenization" itself might split into two or three tokens depending on the vocabulary. This matters in three concrete ways.

First, context windows are measured in tokens, not words. A 128,000-token window holds roughly 96,000 English words, but far less text in languages that the tokenizer handles poorly.

Second, API pricing is per token. Text that tokenizes inefficiently costs more to process. Code with unusual whitespace, non-English text, and emoji all tend to use more tokens per character than plain English prose.

Third, token boundaries affect model behavior in subtle ways. A model sees "unhappiness" as pieces, not as a word, and arithmetic on numbers can behave oddly because digits get grouped into tokens in ways that do not respect place value. Prompt engineers sometimes exploit this, for instance by adding spaces between digits to force digit-by-digit tokenization.

The multilingual gap

Tokenizers are trained on the data the model will see, and that data skews heavily toward English. The result is that English text compresses efficiently while many other languages do not. A sentence in Burmese or Amharic can require several times as many tokens as its English equivalent, which means shorter effective context windows and higher API costs for the same content. This is not a minor inefficiency; it is one of the quiet ways language models serve some languages worse than others. Newer tokenizers trained on more balanced multilingual data narrow the gap, but English still gets the best compression almost everywhere.

Tokenizers are model-specific

Each model family trains its own tokenizer on its own data, and the vocabularies differ. You cannot take token IDs from GPT-4's tokenizer and feed them to Llama; the IDs mean different things. When people quote context window sizes or compare costs across models, the token counts are not directly comparable for exactly this reason. The tokenizer is part of the model, as fixed as its weights.