Language Modeling

LAMBADA

LAnguage Modeling Broadened to Account for Discourse Aspects

Tests whether a model can guess the last word of a passage from a novel when that word can only be predicted from the whole passage.

Released Jun 2016 Metric accuracy (8-shot) SOTA 89.7 · PaLM 540B (2022)
01

At a glance

Task
Last-word prediction
Size
5,153 test passages (plus 4,869 dev); ~2.6M-passage training set from BookCorpus novels
Metric
Accuracy on the target word; also perplexity on it
Construction
Passages kept only if humans guess the word given the whole passage but fail given only the last sentence
Variant
'lambada_openai' — the detokenised version released with GPT-2, now standard
The LAMBADA dataset: Word prediction requiring a broad discourse context
Paperno et al. · ACL 2016
Read the paper ↗ Hugging Face ↗
02

What it tests

LAMBADA is a language-modelling test with a filter. Each passage from a novel ends one word short, and the passage was kept only if people could predict the missing word after reading the whole thing but not from the final sentence by itself.

So the metric rewards exactly the ability that perplexity on random text does not isolate: carrying information across several sentences. A model that only attends to the last few tokens scores near zero.

GPT-2 raised the state of the art from 59% to 63% zero-shot (and perplexity from 99 to 8.6) and noted that most errors were valid continuations that were not the end of a sentence — adding a stop-word filter gave 4 more points. GPT-3 reached 86.4% few-shot by prompting with a fill-in-the-blank format that taught the model to output exactly one word.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Context
"Yes, I thought I was going to lose the baby." "I was scared too," he stated, sincerity flooding his eyes. "You were ?" "Yes, of course. Why do you even ask?" "This baby wasn't exactly planned for."
Target sentence
"Do you honestly think that I would want you to have a ______?"
Answermiscarriage
Context
He shook his head, took a step back and held his hands up as he tried to smile without losing a cigarette. "Yes you can," Julia said in a reassuring voice. "I've already focused on my friend. You just have to click the shutter, on top, here."
Target sentence
He nodded, raising the camera and pointing it at the ______
Answergirl / friend — note: GPT-3 used a 'fill in the blank' prompt with '→' to force a one-word answer
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot (1.5B)63.24 acc8.63 pplPrior SOTA 59.23 acc / 99.8 ppl
Few-shot (15 examples), cloze-style prompt86.4 acc1.92 pplZero-shot 76.2 acc / 3.00 ppl; prior SOTA 68.0
05

State of the art

Best known · accuracy (8-shot)
89.7
open · below human level
Model
PaLM 540B
2022
Human baseline
~86 (original paper's human estimate is informal)

Megatron-Turing NLG 530B 87.2. Rarely reported by frontier models after 2023. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

text (full passage; the target is the final whitespace-separated word). `cimec/lambada` is the original, un-detokenised version.

python · load
from datasets import load_dataset

ds = load_dataset("EleutherAI/lambada_openai", "default", split="test")
print(ds)            # columns: text (full passage; the target is the final whitespace-separated word)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Predict the final word; count it correct only if the whole word is produced greedily
correct = 0
for ex in ds:
    ctx, target = ex["text"].rsplit(" ", 1)
    out = generate(ctx, max_new_tokens=8).strip().split()
    correct += int(out and out[0].strip(".,!?\"'") == target.strip(".,!?\"'"))
print("accuracy", correct / len(ds))
4Or use lm-evaluation-harness

Task name(s): lambada_openai. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks lambada_openai --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Use lambada_openai — the original release has tokenisation artefacts (spaces before punctuation) that depress accuracy.
  • Greedy decoding of the next token is not enough: the target word may be several BPE tokens; score the whole word.
  • Perplexity numbers across papers are not comparable unless the same version and tokeniser were used.
08

References

All benchmarks