Children's Book Test
Tests whether a model can fill a missing word in a sentence from a children's book after reading the twenty sentences before it.
At a glance
What it tests
CBT shows a model 20 sentences of a children's story, then a 21st sentence with one word removed and ten candidate words taken from the context. The question is which candidate fits.
The authors split results by the kind of word removed. Verbs and prepositions are easy for any language model (local syntax decides). Named entities and common nouns need the story — who is in the scene, what object was mentioned — so CN and NE are the numbers people report.
GPT-2 scored each candidate by the probability of the full sentence with that word filled in, reaching 93.3% on CN and 89.1% on NE zero-shot, above the human baseline of ~82%. It has rarely been used since.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Zero-shot (1.5B); candidate scored by full-sentence probability | CN 93.30NE 89.05 | Prior SOTA 85.7 / 82.3; human 81.6 (CN) / 81.6 (NE) |
State of the art
Benchmark effectively retired after GPT-2; later models do not report it. Source: cdn.openai.com ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
sentences (list of 20), question (contains 'XXXXX'), answer, options (10 candidates). Configs: CN, NE, V, P, raw.
from datasets import load_dataset ds = load_dataset("cam-cst/cbt", "CN", split="test") print(ds) # columns: sentences (list of 20), question (contains 'XXXXX'), answer, options (10 candidates) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
acc = 0 for ex in ds: ctx = " ".join(ex["sentences"]) + " " s = [loglik(ctx, ex["question"].replace("XXXXX", o)) for o in ex["options"]] acc += int(ex["options"][s.index(max(s))] == ex["answer"]) print("accuracy", acc / len(ds))
Pitfalls
- Score the full query sentence with each candidate substituted, not just the candidate token.
- Check for overlap with your pre-training data (Project Gutenberg is in most corpora).
- Report CN and NE; V and P are near-ceiling for any LM.
References
- Paper (arXiv 1511.02301)https://arxiv.org/abs/1511.02301
- Hugging Facehttps://huggingface.co/datasets/cam-cst/cbt
- bAbI/CBT downloadhttps://research.facebook.com/downloads/babi/