Language Modeling

One Billion Word

One Billion Word Benchmark (1BW / LM1B)

Measures how well a model predicts the next word in a billion words of shuffled news sentences, with no context between sentences.

Released Dec 2013 Metric test perplexity SOTA 21.8 · Transformer-XL Large (2019)
01

At a glance

Task
Word-level language modelling
Size
~830M training words, 30M sentences; held-out test of ~160k words
Metric
Perplexity
Source
WMT 2011 English news crawl; sentences shuffled, duplicates removed, words seen <3 times → <UNK>
Licence
Apache 2.0
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, Mikolov, Schuster, Ge, Brants, Koehn, Robinson · arXiv 2013 / Interspeech 2014
Read the paper ↗ Hugging Face ↗
02

What it tests

1BW was the first language-modelling dataset big enough to need distributed training. It made large-vocabulary softmax a research problem and was the home turf of ELMo and the Transformer-XL perplexity record.

Its defining quirk is that sentences are shuffled, so there is no cross-sentence context to use. Each sentence stands alone, which is why GPT-2 did worst here (42.2 ppl vs 21.8 SOTA) — its advantage comes from long context, which 1BW throws away.

GPT-3 dropped it entirely because of training-set overlap; it has been largely abandoned since 2019.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Shuffled independent sentences
Three consecutive test lines
Comes with <UNK> drinks and a bag of chips . \n The <UNK> family has been running the farm since 1935 . \n Analysts expect the deal to close in the second quarter .
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot (1.5B)42.16 pplPrior SOTA 21.8 (Transformer-XL); the one LM benchmark GPT-2 lost, blamed on sentence shuffling and heavy preprocessing
05

State of the art

Best known · test perplexity
21.8
no human baseline
Model
Transformer-XL Large (0.8B)
2019
Human baseline
n/a

Benchmark largely abandoned after 2019. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

text (one sentence per row). ~1.8 GB download for train.

python · load
from datasets import load_dataset

ds = load_dataset("billion-word-benchmark/lm1b", split="test")
print(ds)            # columns: text (one sentence per row)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Word-level perplexity with a sliding window (so every token sees up to max_len of context)
import torch, math
text = "\n\n".join(ds["text"]) if "text" in ds.column_names else "\n".join(ds[ds.column_names[0]])
enc = tok(text, return_tensors="pt").input_ids[0]
max_len, stride, nll, n_tok = 1024, 512, 0.0, 0
for start in range(0, enc.size(0), stride):
    end = min(start + max_len, enc.size(0)); trg = end - start if start == 0 else min(stride, end - start)
    ids = enc[start:end].unsqueeze(0).to(model.device); labels = ids.clone(); labels[:, :-trg] = -100
    with torch.no_grad(): nll += model(ids, labels=labels).loss.item() * trg
    n_tok += trg
    if end == enc.size(0): break
n_words = len(text.split())
print("token ppl", math.exp(nll / n_tok), "| word ppl (comparable to GPT-2 table)", math.exp(nll / n_words))
07

Pitfalls

  • Score each sentence independently — that is the benchmark's definition — and count the end-of-sentence token.
  • The test set is in Common Crawl-derived corpora; GPT-3 found >50% overlap and excluded it.
  • '<UNK>' is in the text; byte-level models tokenise it literally.
08

References

All benchmarks