Penn Treebank

Penn Treebank (PTB, Mikolov version)

Measures how well a model predicts the next word in Wall Street Journal text that has been simplified to a 10,000-word vocabulary.

Released 1993 Metric word-level test perplexity (zero-shot) SOTA 20.5 · GPT-3 175B (2020)

At a glance

Task

Word-level language modelling

Size

930k train / 74k valid / 82k test words; 10k vocabulary

Metric

Perplexity

Source

Wall Street Journal 1989; lower-cased, numbers replaced by N, punctuation removed, rare words → <unk>

Licence

LDC (the LM text-only version is widely redistributed)

Building a Large Annotated Corpus of English: The Penn Treebank

Marcus, Santorini, Marcinkiewicz (corpus); Mikolov et al. 2011 (LM split) · Computational Linguistics 1993

Read the paper ↗ Hugging Face ↗

What it tests

PTB is tiny and heavily preprocessed — no capital letters, no punctuation, numbers replaced by 'N', and any word outside the top 10,000 replaced by '<unk>'. For years every recurrent LM paper reported its perplexity because it trains in minutes.

That preprocessing is also why it is a strange test for a modern model: GPT-2 and GPT-3 never saw text that looks like this. Both evaluated it zero-shot anyway (35.8 and 20.5 perplexity), beating models trained on PTB, which says something about how much general language knowledge transfers.

GPT-3 reported PTB and LAMBADA as its only 'traditional' language-modelling results, and explicitly dropped WikiText and 1BW because they overlapped with its training data.

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

What the text looks like

Test text

no it was n't black monday \n but while the new york stock exchange did n't fall apart friday as the dow jones industrial average plunged N points most of it in the final hour it barely managed to stay this side of chaos

Note

'N' replaces every number; '<unk>' appears for rare words.

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

Setting	Reported	Context
Zero-shot (1.5B)	35.76 ppl	Prior SOTA 46.54
Zero-shot (175B)	20.50 ppl	Prior SOTA 35.8 (GPT-2); a 15-point gain

State of the art

Best known · word-level test perplexity (zero-shot)

20.5

no human baseline

Model

GPT-3 175B

2020

Human baseline

n/a

Best models trained on PTB alone are ~50 ppl (dense IndRNN + dynamic eval 49.95). Not reported by any model since GPT-3. Source: arxiv.org ↗ · checked October 2026.

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install

shell

pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn

2Load the data

sentence (one sentence per row, already lower-cased with <unk> and N)

python · load

from datasets import load_dataset

ds = load_dataset("ptb-text-only/ptb_text_only", "penn_treebank", split="test")
print(ds)            # columns: sentence (one sentence per row, already lower-cased with <unk> and N)
print(ds[0])

3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text

python · evaluate

# Word-level perplexity with a sliding window (so every token sees up to max_len of context)
import torch, math
text = "\n\n".join(ds["text"]) if "text" in ds.column_names else "\n".join(ds[ds.column_names[0]])
enc = tok(text, return_tensors="pt").input_ids[0]
max_len, stride, nll, n_tok = 1024, 512, 0.0, 0
for start in range(0, enc.size(0), stride):
    end = min(start + max_len, enc.size(0)); trg = end - start if start == 0 else min(stride, end - start)
    ids = enc[start:end].unsqueeze(0).to(model.device); labels = ids.clone(); labels[:, :-trg] = -100
    with torch.no_grad(): nll += model(ids, labels=labels).loss.item() * trg
    n_tok += trg
    if end == enc.size(0): break
n_words = len(text.split())
print("token ppl", math.exp(nll / n_tok), "| word ppl (comparable to GPT-2 table)", math.exp(nll / n_words))

Pitfalls

Concatenate sentences with newlines before scoring; scoring each sentence independently loses context and inflates perplexity.
'<unk>' and 'N' are literal tokens in the text; a byte-level model will tokenise them as text, which is what GPT-2/3 did.
Word count for normalisation should include the end-of-sentence token if you want to match the classic numbers.

References

Corpus paper (1993)https://aclanthology.org/J93-2004/
Mikolov RNNLM splithttps://www.fit.vutbr.cz/~imikolov/rnnlm/
Hugging Facehttps://huggingface.co/datasets/ptb-text-only/ptb_text_only