enwik8
Measures how well a model predicts raw Wikipedia text one character at a time, scored as bits per character; lower is better.
At a glance
What it tests
enwik8 asks a different question from word perplexity: how many bits does the model need per byte of raw text, markup and all? It is the same quantity a compressor minimises, which is why the Hutter Prize uses it to argue that compression is intelligence.
Because the unit is a byte, there is no vocabulary and no tokeniser argument — every model is scored on exactly the same thing, which makes it one of the fairest LM benchmarks. The price is that sequences are long (100 bytes is a sentence), so it rewards long-context architectures.
GPT-2 reached 0.93 bpb zero-shot on the 1.5B model, matching the SOTA of the time (0.99 bpb from Transformer-XL trained on enwik8).
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Zero-shot (1.5B) | 0.93 bpb | Prior SOTA 0.99 (Transformer-XL) |
State of the art
Expire-Span 0.95, Compressive Transformer 0.97. Research interest moved to larger corpora after 2021. Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
text (single row with the whole 100 MB). Split bytes 0–90M / 90–95M / 95–100M yourself for the standard train/valid/test.
from datasets import load_dataset ds = load_dataset("LTCB/enwik8", split="train") print(ds) # columns: text (single row with the whole 100 MB) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
DATASET = "LTCB/enwik8" import torch, math raw = open("enwik8", "rb").read() if "enwik8" in DATASET else open("text8", "rb").read() test = raw[95_000_000:] # standard split: last 5M bytes text = test.decode("utf-8", errors="replace") enc = tok(text, return_tensors="pt").input_ids[0] nll, max_len, stride = 0.0, 1024, 1024 for i in range(0, enc.size(0) - 1, stride): ids = enc[i:i + max_len].unsqueeze(0).to(model.device) with torch.no_grad(): nll += model(ids, labels=ids).loss.item() * (ids.size(1) - 1) print("bits per byte", nll / (math.log(2) * len(test)))
Pitfalls
- Report bits per *byte*; bpc on the UTF-8-decoded text gives a slightly different number.
- The standard split is by byte offset, not by line.
- For a subword model, convert: bpb = (total NLL in nats) / (ln 2 × number of bytes).
References
- Hutter Prizehttp://prize.hutter1.net/
- Datahttp://mattmahoney.net/dc/textdata.html
- Hugging Facehttps://huggingface.co/datasets/LTCB/enwik8