Tokenization

Unglamorous and directly operational: tokens are your billing unit, your context limit, and the cause of a family of bugs that look like model failures but aren’t.

Why subwords

• Character-level: tiny vocabulary, but sequences become enormous and attention is quadratic.
• Word-level: short sequences, but the vocabulary is unbounded and every unseen word becomes <UNK>.
• Subword: common words stay whole, rare words split into pieces. No out-of-vocabulary case, manageable sequence length.

Subword tokenization is the compromise everything uses.

BPE

Byte-Pair Encoding, the dominant algorithm. Training: start from bytes, repeatedly merge the most frequent adjacent pair, record the merge rules. Encoding: apply the merges greedily.

"tokenization" -> ["token", "ization"]
"antidisestablishmentarianism" -> ["anti", "dis", "establish", "ment", "arian", "ism"]

Byte-level BPE starts from raw bytes rather than characters, so any Unicode input encodes without an unknown token. It’s what GPT-family and most modern tokenizers use.

Alternatives: WordPiece (BERT, merges by likelihood gain rather than raw frequency) and Unigram/SentencePiece (starts large and prunes; handles languages without spaces).

The practical facts

Rules of thumb for English:

1 token ≈ 4 characters ≈ 0.75 words
1,000 tokens ≈ 750 words ≈ 1.5 pages

Never estimate — count:

import tiktoken
enc = tiktoken.encoding_for_model(MODEL)   # use the tokenizer for the model you call
len(enc.encode(text))

Every provider ships a tokenizer. Use the one matching the model you’re calling; counts differ across families and are not interchangeable.

Where tokenization causes real bugs

Non-English text costs more. The same meaning in English, Russian, Chinese or Hindi produces very different token counts — often 2-3x more for non-Latin scripts, because tokenizers are trained on English-heavy corpora and other scripts fragment into many pieces. This is a genuine cost, latency and effective-context-length penalty, and it’s a fairness issue worth naming.

Numbers split unpredictably. 1234 might be one token or three, and the split depends on position and surrounding characters. This is a substantial part of why LLMs are unreliable at arithmetic — the digits aren’t consistently represented. The fix is tools, not prompting.

Trailing whitespace breaks completions. A prompt ending in a space makes the model predict a continuation of that space token, which is off-distribution and degrades output. Strip trailing whitespace from prompts.

Character-level tasks fail. “How many r’s in strawberry” is hard because the model sees a few subword tokens, not letters. It’s a tokenization artefact, not a reasoning failure — and knowing the difference is the interview point.

Structured formats waste tokens. JSON keys, indentation and repeated punctuation all consume budget. Compact JSON, or a terser format, meaningfully reduces cost at volume.

Special tokens

<|begin_of_text|>  <|end_of_text|>  <|eot_id|>
<|start_header_id|>system<|end_header_id|>

Chat models are trained with a specific template of role markers. Getting the template wrong degrades output substantially, and it’s a common failure when calling a model directly rather than through a chat API.

prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": "..."},
     {"role": "user", "content": "..."}],
    tokenize=False, add_generation_prompt=True,
)

Use apply_chat_template rather than hand-assembling the string. It reads the template from the model’s config, so it stays correct across models.

Special tokens are also a prompt injection surface: if user text can contain the literal role markers, it may be able to impersonate a system turn. Tokenizers should encode user content without allowing special tokens — verify this rather than assuming it. See ../14_guardrails_safety/.

Vocabulary size

Typically 32k-256k, trending upward.

Recent models have pushed vocabularies up (128k-256k) specifically to improve non-English efficiency, which partly mitigates the cost disparity above.

Context budgeting

Everything shares one budget: system prompt, tool definitions, retrieved documents, conversation history, and the response.

budget = context_limit - max_output_tokens - len(enc.encode(system + tools))
# what's left is available for history + retrieved context

Two practical rules:

• Reserve output space explicitly. Filling the window with input leaves no room to answer, and the failure is a truncated response rather than a clean error.
• Truncate deliberately. Dropping the oldest turns, summarising them, or re-retrieving are different strategies with different failure modes. See ../12_context_engineering/.

Interview angle

• “Why subword tokenization rather than words or characters?” — words give an unbounded vocabulary and unknown-token failures; characters give sequences too long for quadratic attention. Subwords keep common words whole, split rare ones, and eliminate out-of-vocabulary entirely with byte-level BPE.
• “Why are LLMs bad at arithmetic?” — partly tokenization: numbers split into inconsistent subword pieces depending on position and context, so digits aren’t cleanly represented. Use a calculator tool rather than trying to prompt around it.
• “Why does the same text cost more in Russian than English?” — tokenizers are trained on English-heavy corpora, so other scripts fragment into more tokens. It’s a real cost, latency and effective-context penalty, and larger vocabularies in recent models exist partly to reduce it.
• “How do you count tokens correctly?” — with the tokenizer for that specific model. Counts aren’t portable across model families, and character-based estimates are unreliable for non-English or structured text.
• “A model performs worse when called directly than through the chat API. Why?” — most likely a chat-template mismatch. Use apply_chat_template so role markers match how the model was trained.
• “Any security angle to tokenization?” — special tokens. If user-supplied text can inject literal role markers, it may impersonate a system message. Encode user content with special tokens disallowed.