Building your own GPT-2 tokenizer from scratch

Orientation

What is a tokenizer?

Tokens are the atoms of a language model. Everything it reads and writes is counted in tokens, and the tokenizer is the piece that turns text into tokens and back again.

Tokenization is probably the least fun part of working with LLMs, but it's worth understanding in some detail. A surprising amount of weird model behaviour traces straight back to it. And you may probably learn 1 or 2 things about breaking the model or exploiting its quirks ;)

The idea is simple. You give the tokenizer a sentence. It chops the sentence into chunks which can be sometimes a whole word, sometimes a piece of a word (called sub-word level), or sometimes a single character (called character-level) — and gives each chunk a number. This list of numbers are what the model is going to see:

If you use tokenizer by itself, you will end up creating a sparse matrix of one hot encoded values of a single token. However in practice, the tokenizer is used in conjunction with the embedding table. Inside the model, each number picks out one row of a big table called the embedding table. That row is a list of learned values, and it's what actually goes into the Transformer.

GPT-2 knows 50,257 different chunks (also called the VOCAB SIZE or Vocabulary) and reads up to 1,024 of them at a time. So when people say GPT-2 has a context of 1,024 tokens, they mean 1,024 of these chunks, not 1,024 words or characters.

In the rest of blog, we will build our own GPT-2 tokenizer from scratch and explore how it works in detail.

Motivation

Failures that look like model failures

A lot of problems that look like the model being bad at something are really the tokenizer's fault. Here's the list to keep in mind. We'll come back and explain every one in section 10, once we've built a tokenizer ourselves. As of today most of these problems have been patched by state of the art models providers, however it is still worth looking if you end up building a model of your own.

1. Spelling and reversalWhy can a model that writes fluent prose not count the letters in a word or reverse a string?
2. Non-English textWhy is the same model noticeably worse in Korean or Japanese?
3. ArithmeticWhy is simple addition unreliable? 127 is one token, 677 is two, and four-digit numbers split in arbitrary places.
4. Python under GPT-2Why was GPT-2 so poor at Python compared with GPT-4, beyond the model being smaller?
5. The trailing spaceWhy does the API warn when a prompt ends with a space?
6. The special-token stringWhy does a model stall or refuse when asked to print <|endoftext|>?
7. SolidGoldMagikarpWhy did that string make GPT-3 evasive, insulting or incoherent?
8. YAML vs JSONWhy is YAML cheaper than JSON for the same data?

You can already see the root cause by playing with a single word in tiktokenizer. Depending on whether egg starts a sentence, has a space in front, or is in capitals, GPT-2 turns it into completely different tokens. To the model these are unrelated numbers, and it has to learn from data that they all mean the same thing:

Notice also that the space is part of the token: " egg" and "egg" are different IDs.

The tokenizer also decides how much text fits in the model's context. Higher vocabulary size means same information can be fit into fewer tokens leading to increased context length. For example 300 tokens with GPT-2's tokenizer and worth 185 in GPT-4's. But GPT-4's vocabulary is about twice as big (~100k vs ~50k), and it groups runs of spaces in Python code into one token instead of one token per space. Denser tokens mean the model can look at nearly twice as much text before predicting the next token. A good part of GPT-4's improvement at coding comes from this tokenizer change, not just from the model.

Baseline

The character-level tokenizer

The simplest tokenizer gives every distinct character its own number. It's what you'd build for your first small language model.

Tiny Shakespeare uses 65 different characters, so we make a lookup table from each character to a number from 0 to 64. The embedding table then has 65 rows, one per character.

chars = sorted(set(text))                   # 65 distinct characters
stoi  = {ch: i for i, ch in enumerate(chars)}
itos  = {i: ch for ch, i in stoi.items()}

encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join(itos[i] for i in l)

encode("hii there")                       # [46, 47, 47, 1, 58, 46, 43, 56, 43]

It works, and you get back exactly what you put in. But it has two problems:

• It can't handle new characters. Anything outside those 65 — an accented letter, Korean, an emoji — has no number at all.
• Sequences get long. One token per character means a 1,000-character paragraph takes up 1,000 slots in the context. The context is limited, so the model gets to see very little text at once.

Real models work on chunks, not characters. So we need two things: a way to represent any text, and tokens that hold more than one character. Unicode and UTF-8 solve the first; byte-pair encoding solves the second.

Representation

Why Unicode

In Python, a string is a sequence of Unicode code points. A code point is just the number that the Unicode standard gives to a character.

Unicode covers roughly 150,000 characters across 161 scripts, and it keeps growing with every new version. ord gives you the code point of a character, and chr goes the other way.

ord('H')                              # 104
ord('안')                             # 50504
ord('👋')                             # 128075
[ord(c) for c in '안녕 👋']            # [50504, 45397, 32, 128075]

So we already have a number for any text in any language. Why not just use code points as our tokens? Two reasons:

• The vocabulary would be huge — about 150,000 entries, most of them for characters that almost never show up. Those rows of the embedding table would barely get trained.
• It keeps changing. Unicode is a living standard and adds new characters regularly. A model tied directly to it would have a moving target.

We want something small and fixed that can still represent every character. That's what an encoding gives us.

Representation

Why UTF-8

An encoding turns code points into bytes. There are three of them — UTF-8, UTF-16 and UTF-32 — and UTF-8 is by far the most common.

UTF-8 turns each character into one to four bytes. Simple English letters take one byte; Korean takes three; emoji take four. The first bits of each byte tell you where a character starts and how many bytes it uses:

Why UTF-8 and not the other two?

• It's backwards compatible with ASCII. Plain English text looks exactly the same in ASCII and UTF-8. UTF-16 and UTF-32 don't have this.
• It doesn't waste space. UTF-16 writes "hello" as 68 00 65 00 6C 00 6C 00 6F 00 — lots of zeros. UTF-32 uses four bytes per letter, three of them zero. UTF-8 uses just five bytes.
• There are only 256 possible bytes. Any text in any language becomes a list of numbers between 0 and 255. That's a small, fixed starting vocabulary that never needs to change, no matter what Unicode adds.

list("hello".encode("utf-8"))        # [104, 101, 108, 108, 111]
list("hello".encode("utf-16-le"))     # [104, 0, 101, 0, 108, 0, 108, 0, 111, 0]
list("안녕 👋".encode("utf-8"))        # [236, 149, 136, 235, 133, 149, 32, 240, 159, 145, 139]

So now we can represent any text with just 256 tokens. The catch: sequences get even longer than with characters, because a Korean syllable now takes three slots and an emoji takes four. The embedding table is tiny, but the context fills up with bytes very quickly.

(There is research on feeding raw bytes straight into a model, using a modified Transformer so very long sequences stay affordable. The paper Karpathy points to is MEGABYTE (Meta, 2023), which groups bytes into fixed-size patches: a small model works inside each patch and a big model works across patches. Since then, Meta's Byte Latent Transformer (2024) matched Llama 3's tokenizer-based training up to 8B parameters by making patches bigger where the next byte is easy to predict. H-Net (2025) learns where to draw chunk boundaries itself, and a byte-level H-Net beat a BPE Transformer trained with the same compute. Most production LLMs still use tokenizers, though.)

Algorithm

Byte-pair encoding in one example

Byte-pair encoding (BPE) is an old compression trick, and the whole idea fits in one sentence: find the pair of neighbouring symbols that shows up most often, replace it everywhere with one new symbol, and repeat.

Here's the example from Wikipedia. We have the sequence aaabdaaabac, made from just four letters {a, b, c, d}. It's too long, and we'd like to shorten it:

• Each round makes the sequence shorter and adds exactly one new symbol to the vocabulary.
• New symbols can be merged again. X is made from Z and Y, which were themselves made from the original letters.
• We choose how many rounds to run. Fewer rounds: small vocabulary, long sequences. More rounds: big vocabulary, short sequences.

The algorithm hasn't changed since GPT-2. What changed is how many merges are run, what text they're learned from, and the regex that splits text before merging:

Implementation

Training a tokenizer on Shakespeare

Our training text is Tiny Shakespeare: 1,115,394 characters, all plain ASCII, so also 1,115,394 bytes. We'll do 256 merges, which grows the vocabulary from 256 to 512.

Pre-tokenization: split before merging

If we run BPE on the raw text, it will happily merge across word boundaries. A common word like dog appears next to all kinds of punctuation, so we'd end up with dog., dog! and dog? as three separate tokens — mixing the meaning of the word with punctuation. GPT-2 avoids this by first cutting the text into chunks with a regular expression, and only merging inside each chunk. Two chunks are never merged together.

import regex as re        # pip install regex — the stdlib re lacks \p{L} and \p{N}

gpt2_pattern = re.compile(
    r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""
)

with open('input.txt', encoding='utf-8') as f:
    text = f.read()

chunks = re.findall(gpt2_pattern, text)
ids = [list(chunk.encode('utf-8')) for chunk in chunks]

chunks[:12]  →  ['First', ' Citizen', ':', '\n', 'Before', ' we', ' proceed', ' any', ' further', ',', ' hear', ' me']
len(chunks)  →  297,833

The pattern looks scary, but it's just a list of options separated by |. Going left to right through the text, it grabs the first option that matches:

Note that ids is now a list of lists — one list of bytes per chunk — not one long list. All the functions below work chunk by chunk.

Counting adjacent pairs

get_stats goes through each chunk and counts how often each pair of neighbouring bytes appears. Pairs are never counted across two chunks.

def get_stats(ids):
    count_byte_pairs = {}
    for b in ids:                                        # one chunk at a time
        byte_pairs = [(a, b) for a, b in zip(b[:-1], b[1:])]
        for pair in byte_pairs:
            if pair in count_byte_pairs:
                count_byte_pairs[pair] += 1
            else:
                count_byte_pairs[pair] = 1
    return count_byte_pairs

Replacing a pair

merge walks through a chunk from left to right and replaces every occurrence of pair with the new ID. The i < len(lst) - 1 check makes sure we don't run off the end of the list at the last element.

def merge(lst, pair, idx):
    new_list = []
    i = 0
    while i < len(lst):
        if i < len(lst) - 1 and lst[i] == pair[0] and lst[i+1] == pair[1]:
            new_list.append(idx)
            i += 2                                       # skip both halves
        else:
            new_list.append(lst[i])
            i += 1
    return new_list

merge([5, 6, 6, 7, 9, 1], (6, 7), 99)            # [5, 6, 99, 9, 1]

The training loop

Now we just repeat: count the pairs, pick the most common one, give it a new ID, and replace it everywhere. We keep two dictionaries. merges remembers which pair became which new ID — that's what we'll need for encoding later. vocab remembers which bytes each ID stands for — that's what we'll need for decoding. vocab_pretty is only there so we can print something readable.

VOCAB_SIZE = 512
MERGES = VOCAB_SIZE - 256

vocab        = {i: bytes([i]) for i in range(256)}       # id → bytes
vocab_pretty = {i: chr(i)    for i in range(256)}       # id → printable (ASCII only)
merges       = {}                                        # (id, id) → new id

for k in range(MERGES):
    counts   = get_stats(ids)
    max_pair = max(counts, key=counts.get)
    new_id   = 256 + k

    merges[max_pair]     = new_id
    vocab[new_id]        = vocab[max_pair[0]] + vocab[max_pair[1]]
    vocab_pretty[new_id] = vocab_pretty[max_pair[0]] + vocab_pretty[max_pair[1]]
    print(f"{new_id} : [{vocab_pretty[max_pair[0]]}][{vocab_pretty[max_pair[1]]}] -> [{vocab_pretty[new_id]}]")

    ids = [merge(chunk, max_pair, new_id) for chunk in ids]

256 : [ ][t]    -> [ t]        23,837 occurrences
257 : [h][e]    -> [he]        18,203
258 : [ ][a]    -> [ a]
259 : [o][u]    -> [ou]
260 : [ ][s]    -> [ s]
261 : [ ][m]    -> [ m]
262 : [i][n]    -> [in]
263 : [ ][w]    -> [ w]
264 : [r][e]    -> [re]
265 : [h][a]    -> [ha]
266 : [n][d]    -> [nd]
267 : [ t][he]  -> [ the]       8,684   ← built from two earlier merges
  …
288 : [ y][ou]  -> [ you]
296 : [ a][nd]  -> [ and]
323 : [ t][hat] -> [ that]
414 : [ sha][ll]-> [ shall]
511 : [at][her] -> [ather]

The merge log shows what the tokenizer picked up from Shakespeare:

• Lots of tokens start with a space. The regex sticks the space onto the next word, so "space + first letter of a common word" is the most frequent pair. The tokenizer learns the starts of words before the middles.
• Merges build on each other. 267 is " t" + "he", both made earlier. By 414 we have " shall", and by 511 pieces like "ather" that finish father, rather and gather.
• It depends on the training text. " thou", " thy" and " thee" each get their own token because this is Shakespeare. Trained on Python code, those slots would have gone to runs of spaces instead. The same goes for languages: the more Japanese in the tokenizer's training text, the more Japanese merges, and the shorter Japanese text becomes.

A merge forest. The bottom row is raw bytes. Each merge joins two items into a new one above them, and new tokens can be merged again — 256 and 257 are both children of 267. It's a forest rather than a single tree, because there's no one token at the top that everything merges into.

Implementation

Encoding and decoding

Training gave us a merge table. Now we use it both ways: text to numbers, and numbers back to text. Each direction has one small gotcha.

Decode: IDs → text

For each ID, look up its bytes, join them all together, and decode the result as UTF-8. If we only saved merges, we can rebuild vocab by replaying the merges in the order they were made. (Python dictionaries keep insertion order from 3.7 onwards, so this just works.)

vocab = {i: bytes([i]) for i in range(256)}
for (p0, p1), new_id in merges.items():               # insertion order matters
    vocab[new_id] = vocab[p0] + vocab[p1]

def decode(ids):
    raw = b"".join(vocab[i] for i in ids)
    return raw.decode("utf-8", errors="replace")

decode([97])    →  'a'
decode([128])   →  '�'      # with errors="strict": UnicodeDecodeError, invalid start byte

The gotcha is errors="replace". Not every list of bytes is valid UTF-8. Byte 128 is 10000000 in binary, which doesn't fit any of the patterns in the table in section 5. A model can output any IDs it likes, so it can produce sequences like this. By default Python would crash; with "replace" it puts in a � instead, which is also what OpenAI's code does. If you ever see � in a model's output, it means the model produced tokens that don't spell valid text.

Encode: text → IDs

Split the text into chunks with the same regex, turn each chunk into bytes, then keep merging until nothing more can be merged. The order matters here. Some merges depend on earlier ones (267 needs 256 and 257 to exist first), so at each step we apply the merge that was learned earliest, not the most frequent pair.

def encode(text):
    out = []
    for chunk in re.findall(gpt2_pattern, text):
        toks = list(chunk.encode("utf-8"))
        while len(toks) >= 2:                              # nothing to merge in 0 or 1 tokens
            stats = get_stats([toks])
            pair  = min(stats, key=lambda p: merges.get(p, float("inf")))
            if pair not in merges:                           # every candidate scored inf
                break
            toks = merge(toks, pair, merges[pair])
        out.extend(toks)
    return out

encode("What is up dog? How you doin?")
→ [462, 324, 442, 383, 103, 63, 32, 72, 297, 288, 383, 262, 63]
→ ['What', ' is', ' up', ' do', 'g', '?', ' ', 'H', 'ow', ' you', ' do', 'in', '?']

The trick is in the min line. merges.get(p, float("inf")) gives each pair its merge number, and pairs that were never merged get infinity, so they're never picked. min then picks the earliest merge that applies. When nothing can be merged, every pair scores infinity and min just returns some pair that isn't in merges — that's our signal to stop. The len(toks) >= 2 check handles chunks with a single byte, where there are no pairs at all.

The output tells us something too. " dog" isn't a single token, because 256 merges on Shakespeare never got that far, so it falls back to " do" + "g". "How" at the start of the sentence, with no space in front, becomes "H" + "ow", while " you" in the middle is one token. That's the same egg effect from section 2, showing up in our own little tokenizer.

Round trip

decode(encode(text)) == text                        # True — the full training corpus

val = "The quick brown fox jumps over the lazy dog. 안녕 👋"
decode(encode(val)) == val                          # True — unseen text, unseen script
len(val.encode("utf-8")), len(encode(val))            # (56, 39)

The Korean and the emoji never appeared in Shakespeare, so there are no merges for them and they go through as raw bytes — six tokens for the two Korean syllables, four for the emoji. But they still come back out perfectly. That's the nice thing about starting from bytes: there's no "unknown" token, and nothing ever gets lost.

Encoding then decoding always gives back the original string. The other way round doesn't always work: not every list of IDs is valid UTF-8, so some can't be decoded at all.

Reference

How GPT-2's tokenizer differs from ours

OpenAI's encoder.py is messy code, but it's the same algorithm we just built, with a much bigger vocabulary. Most of the confusion goes away once you match up the names.

A few things the released code doesn't tell you. OpenAI only released the code for using the tokenizer, not for training it. And runs of spaces are never merged in GPT-2 — every indentation space in Python is its own token 220 — so they must have used some extra rule during training that isn't documented anywhere.

GPT-4's tokenizer (cl100k_base, available in OpenAI's tiktoken library) changes the regex: contractions now work in any case, numbers are split into chunks of at most three digits (\p{N}{1,3}), runs of spaces are allowed to merge, and the vocabulary roughly doubles to about 100,000.

Special tokens

On top of the bytes and the merges, you can add special tokens to mark structure in the data. GPT-2 has exactly one: <|endoftext|>, ID 50256. That's why its vocabulary is 50,257 — 256 bytes + 50,000 merges + 1. It goes between documents in the training data, and the model learns that it means "what comes next has nothing to do with what came before".

Special tokens skip BPE entirely: the library looks for the exact string and swaps in the ID. Chat models use lots of them — GPT-3.5-turbo uses <|im_start|> and <|im_end|> to mark where each message starts and ends — and GPT-4 adds a few "fill in the middle" tokens for code.

Motivation, resolved

The failures, explained

Now we can answer every question from section 2. It's really the same answer each time: the model only sees tokens, and the tokens are whatever chunks happened to be common in the tokenizer's training text.

1. Spelling and reversalLong tokens hide their letters. In GPT-4, .DefaultCellStyle is a single token. Ask how many ls it has and the model sees one number, not letters — it answered three (there are four). Ask it to reverse the string and you get a jumble. But ask it to first write the letters out with spaces between them (so each letter becomes its own token) and then reverse that, and it gets it right.
2. Non-English textThe tokenizer was trained mostly on English, so most of its merges are English chunks. "Hello how are you" is 5 tokens; the same sentence in Korean is 15. Everything takes three times as much room in the context. This is about the tokenizer's training text, not just the model's.
3. ArithmeticTo add numbers you work digit by digit. But in GPT-2 a four-digit number might be one token, or two tokens split in any of several ways, depending on what happened to get merged. The model has to learn maths on top of this random chunking. GPT-4 limits number chunks to three digits; Llama 2 splits every digit on its own.
4. Python under GPT-2Every indentation space is a separate token 220. Code ends up being mostly spaces, so the model can fit very little actual code in its context. GPT-4's tokenizer lets runs of spaces merge, which helps a lot before you even touch the model.
5. The trailing spaceIn training text, the space comes at the start of the next word: " Oh" is one token, 8840. If your prompt ends with a space, that space becomes token 220 on its own — something the model has rarely seen. The natural next token " Oh" doesn't fit anymore, because its space has already been used. The model is now in unfamiliar territory, and the output gets worse.
6. The special-token stringSpecial tokens skip BPE: the library spots the exact string and swaps in its ID. If an app does that on user input, typing <|endoftext|> really does insert an end-of-document marker, and the model reacts the way it learned to — by treating everything before it as unrelated. That's also why special tokens are a possible attack surface.
7. SolidGoldMagikarpThis is a Reddit username. The tokenizer was trained on text with a lot of Reddit in it, where this name showed up so often it got its own token. But the model was trained on different text where it barely appeared. So that token's row in the embedding table started out random and was never trained — like unallocated memory in a C program. When you use the word, the model gets a random vector and behaves unpredictably: dodging the question, making things up, even insulting you.
8. YAML vs JSONBraces, quotes and commas each cost tokens. The same data is 116 tokens in JSON and 99 in YAML with GPT-4's tokenizer. You pay per token, both in context space and in money, so YAML is simply cheaper.

Design

Vocabulary size

How many merges to do is the one setting we get to choose. In the model, the vocabulary size shows up in only two places.

class GPT(nn.Module):
    def __init__(self):
        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)   # vocab_size rows
        ...
        self.lm_head = nn.Linear(n_embd, vocab_size)                    # vocab_size logits

A bigger vocabulary means shorter sequences, so the model sees more text at once. So why not make it huge? Three reasons:

• It costs more. Every extra token adds a row to the embedding table and one more output to the last layer, which runs at every position.
• Rare tokens don't get trained. With a million tokens, each one shows up rarely in the training data, so its row barely gets updated — a milder version of SolidGoldMagikarp.
• Too much gets squished into one token. If a single token holds lots of characters, the model gets only one step to make sense of all of them.

In practice, people land somewhere between the high tens of thousands and about 100,000, found by trial and error. Adding tokens to an already-trained model is common too — for example, special tokens for chat or tool use. You just add rows to the embedding table and the last layer, start them off with small random values, and usually train only those new parts while freezing the rest.