WikiText-2
Measures how well a model predicts the next word in Wikipedia articles (small version); lower perplexity means better prediction.
At a glance
What it tests
Perplexity answers: on average, how many words is the model choosing between at each step? A perplexity of 18 means the model is as uncertain as if it were picking uniformly from 18 words. Lower is better.
WikiText fixed two problems with Penn Treebank — it keeps case, punctuation and numbers, and it has a 33k vocabulary instead of 10k — so it is a more realistic test. The -2 version is small enough to train on a laptop; the -103 version is the serious one.
GPT-2 evaluated it zero-shot and reached 18.3 perplexity, versus 39.1 for the prior best (which had been trained on WikiText-2 itself). The number required 'invertible detokenisers' to undo WikiText's preprocessing so GPT-2's byte-level model would see normal text.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Zero-shot (1.5B), with invertible detokeniser | 18.34 ppl | Prior SOTA 39.14 (trained on WikiText-2) |
State of the art
Modern papers (quantisation, pruning) report token-level ppl with each model's own tokeniser — e.g. Llama-3-70B ≈ 2.6 — which is not comparable to word-level numbers. Source: cdn.openai.com ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
text (one line per paragraph; headings start with '='). Use the -raw- configs for byte/subword models.
from datasets import load_dataset ds = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1", split="test") print(ds) # columns: text (one line per paragraph; headings start with '=') print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
# Word-level perplexity with a sliding window (so every token sees up to max_len of context) import torch, math text = "\n\n".join(ds["text"]) if "text" in ds.column_names else "\n".join(ds[ds.column_names[0]]) enc = tok(text, return_tensors="pt").input_ids[0] max_len, stride, nll, n_tok = 1024, 512, 0.0, 0 for start in range(0, enc.size(0), stride): end = min(start + max_len, enc.size(0)); trg = end - start if start == 0 else min(stride, end - start) ids = enc[start:end].unsqueeze(0).to(model.device); labels = ids.clone(); labels[:, :-trg] = -100 with torch.no_grad(): nll += model(ids, labels=labels).loss.item() * trg n_tok += trg if end == enc.size(0): break n_words = len(text.split()) print("token ppl", math.exp(nll / n_tok), "| word ppl (comparable to GPT-2 table)", math.exp(nll / n_words))
Task name(s): wikitext. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks wikitext --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Word-level and token-level perplexity are different numbers; to compare with GPT-2, normalise log-likelihood by the number of *words* in the original text.
- Use a sliding window with a long stride — chunking into independent 1024-token blocks inflates perplexity.
- The non-raw config has '<unk>' tokens; GPT-style models should use raw.
References
- Paper (arXiv 1609.07843)https://arxiv.org/abs/1609.07843
- Hugging Facehttps://huggingface.co/datasets/Salesforce/wikitext
- HF perplexity guidehttps://huggingface.co/docs/transformers/perplexity