Language Modeling

text8

text8

Measures how well a model predicts cleaned-up Wikipedia text (lower-case letters and spaces only) one character at a time.

Released 2006 Metric bits per character (test) SOTA 1.038 · Transformer-XL 24L + dynamic evaluation (2019)
01

At a glance

Task
Character-level language modelling
Size
100M characters; 90M / 5M / 5M split
Metric
Bits per character
Source
enwik9 cleaned: markup removed, digits spelled out, lower-cased, everything but a–z and space dropped
Also used for
word2vec demos
Large Text Compression Benchmark
Matt Mahoney · mattmahoney.net, 2006–
Read the paper ↗
02

What it tests

text8 is the sanitised sibling of enwik8. The same Wikipedia text, but all markup is gone, everything is lower case, and the alphabet is 26 letters plus space. It is the cleanest possible character-level task.

Because there is no markup to exploit, models score slightly worse here than on enwik8 even though the text is simpler — there is less redundancy to compress.

GPT-2 scored 0.98 bpc zero-shot against a SOTA of 1.08 — the only benchmark in its table where the 345M model already beat the prior best.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

What the data looks like
Characters
anarchism originated as a term of abuse first used against early working class radicals including the diggers of the english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way to describe any act that used violent means to destroy the organization of society
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot (1.5B)0.98 bpcPrior SOTA 1.08
05

State of the art

Best known · bits per character (test)
1.038
no human baseline
Model
Transformer-XL 24L + dynamic evaluation
2019
Human baseline
n/a

Transformer-XL alone 1.08. GPT-2's 0.98 is zero-shot and therefore not directly comparable to models trained on text8. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

Download http://mattmahoney.net/dc/text8.zip (31 MB) — a single file of 100,000,000 characters. Not on the Hub in an official form.

shell
curl -LO http://mattmahoney.net/dc/text8.zip && unzip text8.zip   # one 100,000,000-character file
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
DATASET = "text8"
import torch, math
raw = open("enwik8", "rb").read() if "enwik8" in DATASET else open("text8", "rb").read()
test = raw[95_000_000:]                       # standard split: last 5M bytes
text = test.decode("utf-8", errors="replace")
enc = tok(text, return_tensors="pt").input_ids[0]
nll, max_len, stride = 0.0, 1024, 1024
for i in range(0, enc.size(0) - 1, stride):
    ids = enc[i:i + max_len].unsqueeze(0).to(model.device)
    with torch.no_grad(): nll += model(ids, labels=ids).loss.item() * (ids.size(1) - 1)
print("bits per byte", nll / (math.log(2) * len(test)))
07

Pitfalls

  • Same byte-offset split convention as enwik8.
  • Digits were spelled out ('one nine nine two'), which a tokeniser trained on normal text handles oddly.
  • Word-level perplexity on text8 is sometimes reported (with a 44k vocab) — a different number entirely.
08

References

All benchmarks