GPT-2: When Next-Token Prediction Becomes Multitask Learning

The takeaway first

TL;DR

GPT-1 showed that pretraining (training on large number of books) a language model and then fine-tuning it works across many tasks. GPT-2 asks whether a much larger model, trained on a much broader corpus, can begin to do those tasks without any task-specific fine-tuning.

The paper's central bet: broad web text + next-token prediction + scale can produce useful zero-shot behavior on tasks the model was never trained for.

Motivation

From narrow task models to one general language model

NLP systems of the time were built around one labeled dataset and one objective. They did very well on test sets drawn from the same distribution, and broke easily outside it. The authors call these systems "narrow experts", and they are clear about where they think the brittleness comes from:

If narrow training is what hurts generalization, the fix is to train broadly. GPT-2 starts from that ambition: one model should learn enough about language that many tasks can be expressed directly through text, with no dedicated finetuning.

The key observation is that the internet already contains natural demonstrations of many tasks: translation pairs, questions and answers, summaries, definitions and explanations. A model trained only to predict the next token still benefits from learning these patterns, because doing so lowers its loss. A single-task system only learns to map an input to an output. A general system also has to know which task it is being asked to do, and GPT-2's bet is that the task can simply be described in text.

GLUE vs DecaNLP

The paper positions itself against two multitask benchmarks of the time.

Training data

WebText: a deliberately broad corpus

GPT-1 read 7000 books. For GPT-2 the authors went ahead with web pages which contains long contiguous text across domains. They built WebText, a new corpus: the pages behind outbound Reddit links that received at least 3 karma. Reddit votes act as a rough human quality filter. Links that people chose to share and upvote are more likely to hold useful or interesting text. Wikipedia was removed on purpose to avoid overlapping with evaluation datasets.

WebText uses social curation as a filter, then keeps the text behind those links.

Cleaning text with Dragnet and Newspaper

They are web page content extractors. Dragnet extracts article main content and Newspaper also helps with article text along with metadata.

Input representation

Why GPT-2 moved to byte-level BPE

A general language model should be able to assign a probability to any string. And the NLP techniques often used preprocessing steps like lowercasing and stemming, restricting the space of modelable strings. Playing with UNICODE representation of characters was an elegant way to handle this as it contains ~150K characters and can represent any language. However, character level encoding is inefficient. Modeling raw bytes avoids the problem, since every Unicode string can be encoded with only 256 possible byte values, but byte sequences are long and slow to model.

GPT-2 keeps GPT-1's core idea, BPE, but changes what it runs on. GPT-1 cleaned its text with ftfy, split it into words with the spaCy tokenizer, and learned 40,000 merges starting from characters, so a character it never saw in training had no symbol at all. GPT-2 drops the cleanup and the English-specific pre-tokenizer, and runs BPE directly on raw UTF-8 bytes. Any string can now be encoded, and 50,000 merges give a vocabulary of 50,257 tokens (256 bytes + 50,000 merges + 1 end-of-text token). The one rule it adds is that BPE can't merge across character categories, so dog., dog! and dog? don't each take up a vocabulary slot.

Go deeper

Tokenization · GPT-2

Building Your Own GPT-2 Tokenizer from Scratch

Byte-pair encoding over UTF-8, step by step: count pairs, merge, repeat, and train the GPT-2 tokenizer yourself on Shakespeare.

Architecture

GPT-1's architecture, scaled and stabilized

There is no new architecture here. GPT-2 keeps the decoder-only Transformer from GPT-1 and makes a few changes:

• Layer norm moves to the input of each sub-block, and one extra layer norm is added after the final block.
• Initialization changes to stay stable as the stack gets deeper.
• The context window doubles from 512 to 1,024 tokens.
• Batch size grows to 512.

Every size keeps the same head width:

BEFORE BENCHMARKING

A nuance before benchmarking

Before the results, the paper explains how it evaluates GPT-2 on existing language-modeling benchmarks. GPT-2 is still predicting the next token, just on datasets it was never trained on. Two problems come up straight away: every benchmark reports its score in its own unit, and many benchmarks have been preprocessed in ways that look nothing like the web text GPT-2 learned from. The paper puts it like this:

Canonical prediction unit: the benchmark's official unit

GPT-2 splits text into its own BPE tokens, other models split it into words or characters, so "loss per token" means something different for each model. To compare them, every benchmark fixes one official unit (its canonical unit) and everyone divides by that, whatever their tokenizer does internally.

So GPT-2 computes the total log-probability of the whole test set, then divides by the number of canonical units in it:

\(N\)

the number of canonical units (words, characters or bytes) in the benchmark

For example, if a word-level benchmark contains the sentence the cat sat, GPT-2 might spend four BPE tokens on it, but \(N\) is still 3 words. Perplexity is then derived from this per-word loss. Because the denominator is fixed by the benchmark, models with completely different tokenizers can be compared fairly.

Invertible de-tokenizer: undo the benchmark's preprocessing

Many older benchmarks were tokenized before release, with punctuation and contractions split off by spaces:

GPT-2 almost never saw text like that on the web. Some benchmarks even replace rare words with the literal string <UNK>, which appears only 26 times in 40 billion bytes of WebText. The model ends up penalized for not predicting formatting quirks rather than language.

A de-tokenizer is a small script that turns the benchmark text back into natural text. It has to be invertible: there must be an exact, one-to-one way back to the original benchmark format. That way GPT-2 reads natural text, but the probability can still be counted against the official benchmark text and its canonical units, so the score stays comparable. The paper calls this a simple form of domain adaptation, and reports 2.5 to 5 perplexity points of improvement from it.

Experiments · Zero-shot results

Insights from benchmarking

GPT-2 was tested on 8 language-modeling datasets and 5 kinds of downstream task.

Language modeling: how much better than the previous best?

Relative reduction in perplexity (PPL) or bits per character/byte (BPC/BPB) for the 1.5B model versus the previous best. Lower raw scores are better on all of these. Gains are largest where earlier models had the least data or the longest context, and negative on One Billion Word.

• Small datasets gain most. Penn Treebank and WikiText-2 have only 1 to 2 million training tokens, so models trained on them alone had little to learn from. GPT-2 arrives with 40 GB of prior reading.
• One Billion Word goes backwards. Uses destructive preprocessing where sentences are shuffled, which destroys the long-range structure GPT-2 relies on, and it is the largest dataset of the eight.

Downstream tasks: GPT-2 zero-shot vs the reference

All scores on a 0–100 scale, but each task has its own metric, so compare bars within a row, not across rows. The split is the story: tasks that look like next-word prediction are won, tasks that need knowledge or long generation are not.

Benchmark by benchmark

Generalization vs memorization

Did GPT-2 just see the test set?

A zero-shot result can be contaminated if the test examples were already in the training data. With 40 GB of web text, that is a fair worry. The authors check overlap between WebText and public benchmarks using a Bloom filter over normalized 8-grams.

Bloom filters trade exact storage for compact, probabilistic lookup. They can give false positives, never false negatives.

How would you build one?

Normalize the training text, cut it into 8-word shingles (overlapping chunks), hash each shingle into the filter, then query the evaluation shingles with the same normalization.

from pybloom_live import BloomFilter

bf = BloomFilter(capacity=10_000_000, error_rate=0.001)

def ngrams(text, n=8):
    words = text.lower().split()
    return [" ".join(words[i:i+n])
            for i in range(len(words)-n+1)]

for gram in ngrams(training_text):
    bf.add(gram)

"We generated 1M strings, of which zero were found by the filter"

This is a sanity check for false positives. Query the filter with a million strings that should not be in it. Any hit would be a false positive, and they saw none. That does not prove the true rate is zero, only that it is very low.

Table 6: how much of each test set is already in training data?

For six of the language-modeling benchmarks, the paper cuts every test set into 8-grams and checks what percentage of them also appear in a training set. Each column is a benchmark's test set. Each row is the training set it is checked against: the benchmark's own training split, or WebText.

Why are there two rows in Table 6?

The WebText row on its own is hard to judge. Is 3.94% overlap on text8 a lot? The first row gives the answer by providing a baseline: how much a benchmark's test set already overlaps with its own training split. That overlap is normal and accepted, because test and training splits come from the same source and share common phrases, names and boilerplate.

So the real question is whether WebText overlaps with the test sets more than their own training data does. On average it does not: about 3.2% for WebText versus 5.9% for the benchmarks' own training splits. WebText is higher on only two datasets, WikiText-2 and text8, and even there the overlap is a few percent.

Average overlap with test sets. WebText overlap exists, but it is smaller than the overlap already present inside the benchmarks themselves.

Wrapping up

My Learnings

Related reading

GPT-1 · Paper Notes

GPT-1: The Beginning of ChatGPT Revolution

The pretrain-then-fine-tune recipe that GPT-2 set out to simplify.