Language Modeling

WikiText-103

WikiText-103

Measures how well a model predicts the next word in Wikipedia articles (large version with long articles); lower perplexity is better.

Released Sep 2016 Metric test perplexity SOTA 10.81 · Megatron-LM 8.3B (2019)
01

At a glance

Task
Word-level language modelling
Size
103M train / 218k valid / 246k test tokens; 267k vocabulary
Metric
Perplexity
Source
28k Wikipedia Good and Featured articles (same valid/test as WikiText-2)
Licence
CC BY-SA 3.0
Pointer Sentinel Mixture Models
Merity, Xiong, Bradbury, Socher · ICLR 2017
Read the paper ↗ Hugging Face ↗
02

What it tests

WikiText-103 is the same validation and test text as WikiText-2 but with fifty times more training data. Because articles are long and the test set keeps them intact, it rewards models that can use long context — it was the benchmark where Transformer-XL, adaptive softmax and compressive memory made their names.

GPT-2 reached 17.5 perplexity zero-shot, close to the then-SOTA of 18.3 that had been trained on the dataset. Megatron-LM (8.3B) later pushed it to 10.8, and retrieval models (RETRO) reported far lower numbers with the caveat that their retrieval database overlapped with the test set.

Today almost nobody trains on WikiText-103 specifically, so the leaderboard stopped around 2021.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Same format as WikiText-2
Raw
= Robert Boulter = \n\n Robert Boulter is an English film , television and theatre actor . He had a guest @-@ starring role on the television series The Bill in 2000 .
Note
Test set is ~245k words in ~60 articles.
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot (1.5B)17.48 pplPrior SOTA 18.3 (trained on WikiText-103)
05

State of the art

Best known · test perplexity
10.81
no human baseline
Model
Megatron-LM 8.3B
2019
Human baseline
n/a

RETRO 7.5B reports 2.4 with retrieval, but with acknowledged train/test overlap in the retrieval DB. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

text. ~540 MB download for the raw train split; the test split alone is small.

python · load
from datasets import load_dataset

ds = load_dataset("Salesforce/wikitext", "wikitext-103-raw-v1", split="test")
print(ds)            # columns: text
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Word-level perplexity with a sliding window (so every token sees up to max_len of context)
import torch, math
text = "\n\n".join(ds["text"]) if "text" in ds.column_names else "\n".join(ds[ds.column_names[0]])
enc = tok(text, return_tensors="pt").input_ids[0]
max_len, stride, nll, n_tok = 1024, 512, 0.0, 0
for start in range(0, enc.size(0), stride):
    end = min(start + max_len, enc.size(0)); trg = end - start if start == 0 else min(stride, end - start)
    ids = enc[start:end].unsqueeze(0).to(model.device); labels = ids.clone(); labels[:, :-trg] = -100
    with torch.no_grad(): nll += model(ids, labels=labels).loss.item() * trg
    n_tok += trg
    if end == enc.size(0): break
n_words = len(text.split())
print("token ppl", math.exp(nll / n_tok), "| word ppl (comparable to GPT-2 table)", math.exp(nll / n_words))
4Or use lm-evaluation-harness

Task name(s): wikitext. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks wikitext --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Same word-vs-token perplexity caveat as WikiText-2.
  • Articles are concatenated; evaluate with a sliding window over the whole test file rather than per line.
  • The test set is public and in Common Crawl — expect contamination for any web-trained model.
08

References

All benchmarks