OpenBookQA
Tests whether a model can combine a given science fact with everyday knowledge to answer a question.
At a glance
What it tests
The 'open book' is a list of 1,326 elementary science facts ('metal is a thermal conductor'). Each question was written to require one of those facts plus something everybody knows but nobody wrote down ('a suit of armour is made of metal').
That combination — retrieve a core fact, then reason with background knowledge — is what makes it harder than its size suggests. Retrieval alone cannot answer it, and neither can a model that has only memorised the book.
GPT-3 scored 65.4% few-shot. The human baseline of 91.7% has since been passed by fine-tuned systems.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Few-shot (100 examples) | 65.4 acc | Zero-shot 57.6; fine-tuned SOTA then 87.2 (UnifiedQA) |
State of the art
GPT-4 91.0; approximate (leaderboard value as reproduced in secondary papers). Source: leaderboard.allenai.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
question_stem, choices (text, label), answerKey. The 'additional' config adds fact1 (the book fact) and humanScore.
from datasets import load_dataset ds = load_dataset("allenai/openbookqa", "main", split="test") print(ds) # columns: question_stem, choices (text, label), answerKey print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
# Science MC: score each option text as a continuation of "Question: … Answer:" acc = 0 for ex in ds: q = ex.get("question") or ex["question_stem"] texts, labels = ex["choices"]["text"], ex["choices"]["label"] s = [loglik(f"Question: {q}\nAnswer:", " " + t) / max(1, len(t)) for t in texts] acc += int(labels[s.index(max(s))] == ex["answerKey"]) print("acc_norm", acc / len(ds))
Task name(s): openbookqa. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks openbookqa --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Only 500 test questions — one question is 0.2 points.
- Giving the model the book fact (the 'additional' config) changes the task; say whether you did.
- Question stems are sometimes incomplete sentences that the option completes; scoring option text as a continuation works well.
References
- Paper (arXiv 1809.02789)https://arxiv.org/abs/1809.02789
- AI2 leaderboardhttps://leaderboard.allenai.org/open_book_qa/
- Hugging Facehttps://huggingface.co/datasets/allenai/openbookqa