Send "Hello, world!" to GPT-4 and you pay for 4 tokens. Send the exact same string to Claude and the bill is for 3. Same words, different math, and the reason is a 1994 data compression trick called byte pair encoding, or BPE, that quietly decides how much every AI conversation costs.
BPE is the algorithm that turns your text into the numbers a model actually reads. It sits underneath GPT, Claude, Llama, Mistral, and most open weight models you have probably reviewed on this site. It also quietly shapes your context window, your API bill, and how well a model handles a name, a typo, or a language other than English. Understand it once and token limits, weird pricing jumps, and strange model behavior on rare words all stop feeling random.
This is a deep, complete pass at the topic. We will build a BPE vocabulary by hand, walk through real production vocabulary sizes across every major model family, look at where the algorithm actually breaks, and cover the research trying to replace it in 2026.
1. What Is Byte Pair Encoding (BPE)?
Byte pair encoding is a subword tokenization algorithm that builds a vocabulary by repeatedly merging the most frequent pair of adjacent symbols in a text corpus, starting from individual characters or bytes and working upward until it reaches a target vocabulary size. It was not invented for AI at all. Philip Gage described the original byte pair encoding technique in 1994 as a general purpose data compression method, built to replace the most common byte pair in a file with a single unused byte, and repeat.
Rico Sennrich, Barry Haddow, and Alexandra Birch adapted the compression idea for language models in their 2016 paper, "Neural Machine Translation of Rare Words with Subword Units", presented at the Association for Computational Linguistics conference in Berlin. Their insight was that the same merge-the-most-frequent-pair logic used for compression could build an open vocabulary for translation, one that never hits an unknown word wall, by encoding rare words as sequences of smaller, already-known subword units. In 2019, OpenAI researcher Alec Radford and colleagues pushed the idea one step further for GPT-2, switching the base units from characters to raw bytes, which is the byte-level BPE that every major LLM tokenizer descends from today.
Instead of assigning one number per word, which would need a vocabulary in the hundreds of thousands and still choke on typos and new slang, BPE breaks text into subword pieces. Common words like "the" stay as one token. Rarer words like "tokenization" get split into pieces like "token" and "ization". If you want to see the full mechanics of how these token counts get spent inside a production model, Tiktoken: High-Performance Tokenizer for OpenAI Models walks through OpenAI's implementation of exactly this idea.
Quotable: BPE does not know what words mean. It only knows which byte sequences showed up together most often in its training data, and it merges on frequency, not grammar.
2. Why Tokenization Exists: The Word-Level Vocabulary Problem
Tokenization exists because word-level vocabularies do not scale and cannot handle text a model has never seen before. English alone has over 170,000 words in current active use. Add technical jargon, brand names, typos, code identifiers, and every other language a model needs to support, and a naive word-per-token vocabulary balloons into the millions.
Two problems follow directly from that. First, the embedding matrix, the lookup table that turns each vocabulary entry into a vector the model can compute with, grows linearly with vocabulary size. A vocabulary of 2 million words at a modest 4,096-dimensional embedding would need roughly 8 billion parameters just to store word identities, before the model has done a single unit of actual reasoning. Second, any word not already in that fixed vocabulary becomes an unknown token, typically written as <unk>, which is a dead end for the model: it cannot spell the word, translate it, or reason about it, only shrug.
Subword tokenization solves both problems at once. A vocabulary of 50,000 to 250,000 subword pieces, tiny compared to a word-level vocabulary, can still represent effectively unlimited text, because any unseen word can always be broken down into smaller pieces the tokenizer has seen: common words stay whole, rare words split into meaningful fragments, and worst case, into raw bytes. Nothing is ever truly unknown.
This tradeoff, vocabulary size against sequence length against embedding table cost, is one of the most underrated architecture decisions a lab makes, and it interacts directly with ideas covered in attention mechanism in LLMs explained, since every token the tokenizer produces becomes one more position the attention mechanism has to relate to every other position in the sequence.
3. How BPE Training Works: The Merge Algorithm, Step by Step
BPE training runs in four repeating steps: split text into characters, count every adjacent pair, merge the single most frequent pair into a new symbol, and repeat until you hit a target vocabulary size. Each merge gets recorded, in order, into a permanent merge list. That ordered list is the tokenizer. Here is the classic toy example used to teach it, worked through in full.
Start with a tiny corpus containing four words and their frequencies: low (5), lower (2), newest (6), widest (3). Split every word into characters plus an end-of-word marker, written here as an underscore:
l o w (x5) l o w e r (x2) n e w e s t (x6) w i d e s t (x3)Now count every adjacent character pair across the whole corpus, weighted by word frequency. The pair "e" and "s" appears in "newest" (6 times) and "widest" (3 times), for a combined count of 9, the highest of any pair. Merge "e" and "s" into a new symbol "es" and rewrite the corpus:
l o w (x5) l o w e r (x2) n e w es t (x6) w i d es t (x3)Recount. Now "es" and "t" is the most frequent pair at 9 occurrences. Merge into "est". Recount again: "est" and "_" is now most frequent, merge into "est_". The loop continues, and after several more rounds a vocabulary emerges that looks roughly like this:
l, o, w, e, r, n, s, t, i, d, es, st, est, est_, low, low_, new, newest_Each row in this progression is one merge rule, and the full ordered list of merge rules, applied in the exact sequence they were learned, is what a tokenizer actually is. There is no dictionary lookup at inference time in the way a spellchecker works. The tokenizer replays its learned merges, in order, against new text every single time.
A minimal training loop for this algorithm is short enough to write from scratch. In pseudocode:
vocab = set(all_bytes_or_chars_in_corpus) merges = [] while len(vocab) < target_vocab_size:
pair_counts = count_adjacent_pairs(corpus)
best_pair = argmax(pair_counts)
corpus = replace_pair_with_new_token(corpus, best_pair)
vocab.add(best_pair)
merges.append(best_pair)Real implementations add speed tricks, a max-heap for pair counts instead of a full recount every iteration, careful handling of word boundaries, and Unicode normalization, but the core loop above is the entire idea. If you want to run this yourself against a real corpus and compare it to a production tokenizer's output, the Build Fast with AI gen-ai-experiments cookbook repository has hands-on notebooks for working through Hugging Face's tokenizer stack from first principles.
At inference time, the merge rules are applied deterministically and in the exact order they were learned. That is why the split of a word depends on training data frequency, not English spelling rules. "Predict" might tokenize as ["pre", "dict"] while "prediction" splits as ["predict", "ion"], because those specific pairs happened to be common enough in the training corpus to earn separate merges. Two tokenizers trained on different corpora, even with identical vocabulary sizes, will tokenize the same sentence differently, because the merge order itself encodes the statistics of whatever text the tokenizer was trained on.
4. Byte-Level BPE: Why Modern LLMs Tokenize Bytes, Not Characters
Every major LLM tokenizer today uses byte-level BPE, meaning the starting vocabulary is the 256 possible byte values rather than a language-specific character set. Sennrich's original 2016 formulation used characters as the base unit, which meant any character outside the training alphabet still produced an unknown token. Radford and OpenAI's 2019 GPT-2 paper switched the base unit to raw UTF-8 bytes, which guarantees every possible string, in any language, including emoji and malformed text, can always be represented, since any Unicode character can be decomposed into bytes, and every byte value is already in the base vocabulary.
This single change is why modern tokenizers essentially never produce an unknown token, unlike older word-level and even early character-level systems. The tradeoff is that byte-level BPE needs a regex-based pre-tokenization pass before the merge algorithm ever runs, splitting raw text into chunks, roughly matching words, numbers, and punctuation groups, so that merges never accidentally span across a word boundary into unrelated text. GPT-2's pre-tokenization regex handles contractions, letter runs, number runs, and whitespace runs as separate chunk types, and BPE then trains and applies independently within each chunk.
Without this pre-tokenization step, a tokenizer could learn a merge that spans from the end of one word into the start of the next, producing meaningless tokens like a fragment ending in one word and starting the next. The regex pass is what keeps every learned merge meaningful and word-boundary-respecting, even though the underlying algorithm operates purely on byte frequency with no concept of what a word is.
5. BPE vs WordPiece vs Unigram vs SentencePiece
BPE is the most widely used subword algorithm in modern LLMs, but it is not the only one, and the differences matter for anyone choosing or training a tokenizer. WordPiece, the algorithm behind BERT, DistilBERT, and ELECTRA, merges the pair that most increases the likelihood of the training data rather than the pair with the raw highest frequency, a subtly different scoring rule that tends to favor merges that are more informative even if slightly less common. Unigram, used by T5, ALBERT, and mBART, works in the opposite direction entirely: it starts from a large candidate vocabulary and iteratively prunes tokens whose removal causes the smallest increase in corpus loss, rather than building up from nothing.
SentencePiece is not a competing algorithm at all. It is a library that can implement either BPE or Unigram directly on a raw byte or character stream, with no assumption that whitespace separates words. That last detail matters enormously for Chinese, Japanese, and Thai, which do not use spaces between words the way English does. SentencePiece encodes the space character itself as a visible metasymbol in its vocabulary, commonly written as an underscore-like glyph, which lets the same algorithm handle space-delimited and non-space-delimited languages with one consistent approach.
An important nuance: Unigram is probabilistic where BPE is deterministic. Given a word like "hugs", BPE always produces the exact same split every time, because it replays a fixed merge list in a fixed order. Unigram can consider several candidate segmentations, such as ["hug", "s"] or ["h", "ug", "s"], and picks the one with the highest probability under its learned model, which means it can sample different tokenizations of the same word during training, a technique called subword regularization that BPE cannot natively do.
If you are choosing a tokenizer to pair with a retrieval pipeline, this choice interacts with chunking strategy more than most teams assume, and it pairs closely with the architecture ideas in what is Mixture of Experts (MoE) and how it works, since both tokenizer and routing architecture are decisions made once at training time and then locked in for the entire life of a model family.
Don't just use ChatGPT. Learn to build custom LLM agents, RAG pipelines, and full-stack Agentic AI apps in our intensive 6-week program.
6. Which Tokenizer Powers GPT, Claude, Gemini, and Llama in 2026
GPT-4 and GPT-3.5-Turbo use OpenAI's cl100k_base encoding at roughly 100,000 tokens, while GPT-4o, the o-series, and the GPT-5 family moved to o200k_base at roughly 200,000 tokens, a jump that shrinks token counts on code and non-English text by ten to twenty percent on the same input, since the larger vocabulary has room for far more code-friendly and multilingual merges. Claude exposes a count-tokens API endpoint rather than a public offline encoder, and Anthropic has kept the underlying merge tables closed since Claude 3, a change from Claude 1 and Claude 2, which shared an open source tokenizer.
Gemini and Gemma use Google's SentencePiece implementation with a documented vocabulary of around 256,000 tokens, the largest of the major closed models, which is part of why Gemini tends to post the best fertility scores, meaning fewer tokens per word, on non-English benchmarks. Llama made a particularly large jump between generations: Llama 2 used a 32,000-token SentencePiece BPE vocabulary, while Llama 3 switched to a tiktoken-style byte-level BPE tokenizer with 128,256 tokens, roughly four times larger, packing meaningfully more text into each token and substantially improving efficiency on both code and non-English scripts.
Contrarian point worth saying plainly: vocabulary size comparisons across providers get thrown around online as a quality signal, and they should not be. A bigger vocabulary means fewer tokens for the same text, which affects your bill and your effective context window, but it says nothing on its own about reasoning ability, benchmark performance, or model quality. Gemini's 256,000-token vocabulary does not make it a better reasoner than Claude's undisclosed one; it makes non-English and code text cheaper to represent.
Same string, three vendors, three different token counts, and the gap widens further once KV cache in LLMs enters the picture, since every token you save at the tokenizer level is one less slot that cache has to hold and one less position the model has to attend over during generation.
7. The Real Cost of Tokens: Vocabulary Size, Pricing, and Context Windows
Tokenizer choice has a direct, measurable effect on your API bill, because every provider charges per token, and a tokenizer that needs more tokens to represent the same text costs more money for identical content. In English, one token is roughly 4 characters or about 0.75 words as a rule of thumb, so 1,000 words of English typically becomes 1,300 to 1,500 tokens depending on the exact tokenizer and vocabulary in use.
That ratio degrades sharply outside English. On cl100k_base, Japanese, Chinese, and Korean text can run at close to one token per character, roughly four times worse than English on the same encoding, because the tokenizer's training corpus and merge list were skewed toward English and Latin-script text. Code has its own cost profile: whitespace, indentation, and newlines alone consume roughly 24.5 percent of tokens across programming languages while contributing almost no semantic value, and this overhead compounds further when a model outputs structured formats like JSON, where braces, quotes, and indentation stack on top of the actual content.
This is also why context window numbers advertised by different providers are not directly comparable in practical terms. A 128,000-token context window on a tokenizer with a 256,000-word vocabulary holds meaningfully more actual English or Japanese text than the same numeric limit on a smaller-vocabulary tokenizer, because each token represents more raw content on average. Anyone estimating how much of a document fits in context should test with the target model's actual tokenizer, not assume token counts translate one-to-one across providers.
8. Where BPE Breaks: Glitch Tokens and Language Bias
BPE breaks down when a token earns a slot in the vocabulary through pure frequency in the tokenizer's training corpus but almost never appears in the model's actual training data, producing what researchers call a glitch token. The most famous case, discovered by Jessica Rumbelow and Matthew Watkins in January 2023, found that asking early ChatGPT to repeat the string "SolidGoldMagikarp", a Reddit username frequent enough in the tokenizer's corpus to earn its own dedicated BPE token, produced the unrelated word "distribute" instead, because the embedding vector for that token was essentially untrained noise, never meaningfully updated during model training despite having its own vocabulary slot.
The problem has not gone away. Systematic research in 2026 found that roughly 4.3 percent of vocabulary entries across tested models qualify as glitch tokens, and the GlitchMiner framework, presented at AAAI 2026, uses gradient-based entropy maximization to hunt them down across GPT-4, Llama 2, Mistral, and DeepSeek-V3. The root cause is always the same mismatch: the tokenizer is trained on one corpus, the model is trained on a different, related but not identical corpus, and any token frequent in the first but rare in the second ends up with a vocabulary slot and an essentially random embedding.
Language bias is the second major limitation, and it is a data problem rather than an algorithm flaw. Tokenizers trained mostly on English text charge other languages more tokens for the same meaning, sometimes running close to one token per character on Japanese, Chinese, or Korean text. Research on tokenization efficiency, or fertility, across foundational LLMs for lower-resource languages like Ukrainian found that closed models can reach up to 4.4 tokens per word on narrow technical topics, with the smallest performance degradation showing up in models with the largest, most multilingual vocabularies, exactly the tradeoff described in the vocabulary size section above.
Frozen vocabulary is the third practical limitation worth naming directly. Once a tokenizer finishes training, its merge list is locked. A brand name, a piece of tech slang, or a term that did not exist during tokenizer training gets chopped into whatever subword pieces the frozen vocabulary happens to offer, which is often wasteful but never broken, since byte-level BPE can always fall back to individual bytes as a last resort.
9. Building Your Own Tokenizer: What to Know Before You Train One
Training a custom tokenizer makes sense when your domain uses specialized vocabulary, medical terminology, legal language, chemical formulas, or dense code identifiers, that a general-purpose tokenizer splits into many wasteful subwords. A tokenizer trained on your own domain-specific corpus will assign single tokens to terms a general tokenizer might break into three or four pieces, which shortens sequences, lowers cost, and can measurably improve downstream fine-tuning performance.
The practical path most teams take is the Hugging Face tokenizers library, a Rust-backed implementation supporting BPE, WordPiece, and Unigram with full training support, or Andrej Karpathy's minbpe project, a minimal, readable reference implementation built specifically for learning how the algorithm works internally rather than for production speed. Both let you train a BPE vocabulary on your own corpus in well under an hour on a laptop, for vocabulary sizes in the tens of thousands.
Before training your own, run the smallest possible sanity check: take a representative sample of your actual domain text, tokenize it with a general-purpose tokenizer like cl100k_base, and count how many tokens the specialized terms in your domain consume compared to common English words of similar length. If domain terms are consistently costing three to four times more tokens than equivalent everyday words, that is a strong, concrete signal that a custom tokenizer is worth the training time. The Build Fast with AI gen-ai-experiments cookbook repository has starter notebooks that walk through exactly this kind of tokenizer evaluation before you commit to training your own from scratch.
One caution worth stating plainly: a custom tokenizer only helps if you are also training or fine-tuning the model that uses it. Swapping tokenizers on a model whose weights were trained against a different vocabulary requires re-training the embedding layer at minimum, and often more, so this is a decision made early in a project, not a drop-in swap on a model you are only calling through an API.
10. What's Next After BPE: SuperBPE, BoundlessBPE, and LiteToken
Standard BPE has a built-in ceiling because it refuses to merge tokens across whitespace boundaries, a constraint inherited from the regex pre-tokenization step covered earlier. Two 2025 research papers presented at COLM went after that constraint directly. SuperBPE runs a standard BPE pass first, then learns a second pass of cross-word superword tokens that can span whitespace entirely, producing 33 percent fewer tokens and a 4.0 percent average performance gain across 30 benchmarks, including an 8.2 percent gain on MMLU, while winning on 25 of the 30 individual tasks tested, and training in a matter of hours on 100 CPUs.
BoundlessBPE goes further still and removes the whitespace boundary constraint entirely, allowing merges across word boundaries from the very start of training rather than as a second pass. It reports up to a 15 percent improvement in bytes per token along with a 3 to 5 percent gain in encoding efficiency, measured by Renyi efficiency, over standard BPE on comparable vocabulary sizes.
A separate, more surgical February 2026 technique called LiteToken takes a cleanup approach rather than a redesign. It identifies and removes intermediate merge residues, tokens that appear frequently during the BPE training process itself but rarely survive into the final tokenized output of real text, shrinking a trained vocabulary's practical footprint without needing to retrain the entire tokenizer from scratch.
None of this changes the fundamentals covered in LLM scaling laws explained, but it does mean the tokenizer layer, long treated as a solved, boring preprocessing step that nobody needed to think about after 2019, is now an active research area in its own right, with real, measured accuracy gains on the table rather than only compression gains. Expect at least one major model family to ship a production tokenizer built on one of these ideas before the end of 2026.
Frequently Asked Questions
What is byte pair encoding in NLP?
Byte pair encoding is a subword tokenization algorithm that builds a vocabulary by iteratively merging the most frequent adjacent pair of characters or bytes in a training corpus, originally developed as a 1994 compression technique by Philip Gage and adapted for NLP by Sennrich, Haddow, and Birch in their 2016 ACL paper.
How does BPE tokenization work?
BPE starts with individual characters or bytes, counts every adjacent pair across a training corpus, merges the most frequent pair into a new token, and repeats this loop until the vocabulary reaches a target size, typically tens to hundreds of thousands of tokens for modern LLMs, recording every merge in a fixed, ordered list.
What is the difference between BPE and WordPiece?
BPE merges the most frequent adjacent pair by raw count, while WordPiece, used by BERT and DistilBERT, merges the pair that most increases the likelihood of the training data, a subtly different scoring rule that can produce different subword splits on identical text.
Why do LLMs use tokens instead of words?
Word-level vocabularies would need hundreds of thousands to millions of entries to cover every possible word, name, and typo, and any unseen word becomes an unusable unknown token. Subword tokens like BPE keep the vocabulary compact, typically 50,000 to 250,000 entries, while still covering unseen text by falling back to smaller known pieces, and byte-level BPE specifically guarantees no text is ever truly unrepresentable.
What tokenizer does GPT-4 use?
GPT-4 and GPT-3.5-Turbo use OpenAI's cl100k_base BPE encoding at roughly 100,000 tokens, while GPT-4o and the GPT-5 family moved to the larger o200k_base encoding at roughly 200,000 tokens for better efficiency on code and non-English text.
Does Claude use BPE?
Claude 1 and Claude 2 shared an open source BPE tokenizer, but Anthropic switched to a proprietary, closed tokenizer starting with Claude 3, exposed to developers only through a count-tokens API endpoint rather than a public offline encoder.
How many tokens is a word on average?
In English, one token is roughly 4 characters or about 0.75 words, so 1,000 words typically becomes 1,300 to 1,500 tokens, though the exact ratio depends on the specific tokenizer and vocabulary size, and can run several times higher for non-English scripts or dense code.
What are glitch tokens and why do they happen?
Glitch tokens are vocabulary entries that were frequent enough in a tokenizer's training corpus to earn a dedicated token, but rare enough in the model's actual training data that their embedding was never meaningfully learned, causing the model to produce unrelated or broken output when that token appears, as seen in the well-documented SolidGoldMagikarp case from January 2023.
Recommended Blogs
ā What Is Tiktoken (OpenAI Tokenizer)
ā What Is Mixture of Experts (MoE)
ā Attention Mechanism in LLMs Explained
ā LLM Scaling Laws Explained
ā What Is RLHF, LLM Training Guide
Resources & Community
Join our community of 70,000+ AI enthusiasts and learn to build powerful AI applications! Whether you're a beginner or an experienced developer, Build Fast with AI helps you understand and implement AI in your projects.
ā Website, buildfastwithai.com
ā LinkedIn, Build Fast with AI
ā Instagram, buildfastwithai
Agentic AI Launchpad 2026
A structured 6-week cohort program that takes you from AI basics to building and deploying real-world agentic AI systems. Includes live sessions, expert mentorship, project reviews, and a builder community network.
Ready to go from learning to building? Join the next cohort: Agentic AI Launchpad 2026
Free AI Resources
Access free tools, workshops, and micro-learning to keep building:
ā AI Workshops, free resources and past recordings
ā Unrot, learn AI in 5 minutes a day
Tokenization is the layer nobody talks about until the bill arrives. Follow Build Fast with AI for more LLM internals explained in plain English, no fluff.
References
ā OpenAI, tiktoken BPE tokenizer
ā Hugging Face, tokenization algorithms guide
ā Sennrich Haddow Birch, subword-nmt reference implementation
ā Let's Data Science, tokenization deep dive 2026
ā InventiveHQ, LLM tokens explained
ā Dev Community, tokenizer quirks across Claude GPT Gemini
ā Machine Learning Plus, build a BPE tokenizer from scratch
ā arXiv, data mixture inference from BPE tokenizers
Frontiers, tokenization efficiency for the Ukrainian language