Reading Comprehension

SQuAD 2.0

Stanford Question Answering Dataset v2

Tests whether a model can find the answer to a question in a Wikipedia paragraph, or recognise when the paragraph does not contain it.

Released Jun 2018 Metric test SOTA 93.2 F1 / 90.9 EM · IE-Net ensemble (2021)
01

At a glance

Task
Extractive span or no-answer
Size
130k train / 12k dev / 9k test (hidden); ~33% unanswerable in dev
Metric
Exact match and F1
Source
Wikipedia paragraphs; unanswerable questions written by crowd-workers to resemble answerable ones
Predecessor
SQuAD 1.1 (2016), 100k answerable questions
Know What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, Jia, Liang · ACL 2018
Read the paper ↗ Hugging Face ↗
02

What it tests

SQuAD 1.1 was the dataset that made extractive QA a mainstream task: read a Wikipedia paragraph, highlight the span that answers the question. Models passed human performance on it within two years, largely because every question had an answer and the answer was always a span.

SQuAD 2.0 adds questions that are plausible but unanswerable from the paragraph — same entities, same wording style, but the paragraph does not actually say. A model now has to decide whether to answer at all, which is a very different skill from locating a span.

GPT-3 evaluated it few-shot and reached 69.8 F1, far below fine-tuned systems. The benchmark has since been saturated by fine-tuned ensembles.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Answerable
Paragraph (excerpt)
The Normans (Norman: Nourmands; French: Normands) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France…
Question
In what country is Normandy located?
AnswerFrance
Unanswerable (adversarially written)
Paragraph (excerpt)
…Later the Normans spread to England, Southern Italy and Sicily…
Question
Who gave their name to Normandy in the 1000s and 1100s?
AnswerNo answer (the paragraph says 10th–11th centuries, not 1000s–1100s as written — trick question)
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (16 examples)69.8 F1Zero-shot 59.5; fine-tuned SOTA then 93.0
05

State of the art

Best known · test
93.2 F1 / 90.9 EM
✓ saturated · at or above human level
Model
IE-Net ensemble (RICOH)
2021
Human baseline
89.5 F1 / 86.8 EM

Source: rajpurkar.github.io ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

context, question, answers (text list, answer_start list — empty lists mean unanswerable)

python · load
from datasets import load_dataset

ds = load_dataset("rajpurkar/squad_v2", split="validation")
print(ds)            # columns: context, question, answers (text list, answer_start list — empty lists mean unanswerable)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
import evaluate
metric = evaluate.load("squad_v2")           # for DROP use the official drop_eval.py from allenai/allennlp-reading-comprehension
preds, refs = [], []
for ex in ds:
    p = f"{ex['context'] if 'context' in ex else ex['passage']}\nQuestion: {ex['question']}\nAnswer (or 'unanswerable'):"
    out = generate(p, max_new_tokens=32, stop="\n").strip()
    preds.append({"id": ex["id"], "prediction_text": "" if out.lower().startswith("unanswerable") else out, "no_answer_probability": float(out.lower().startswith("unanswerable"))})
    refs.append({"id": ex["id"], "answers": ex["answers"]})
print(metric.compute(predictions=preds, references=refs))
4Or use lm-evaluation-harness

Task name(s): squadv2. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks squadv2 --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Report HasAns and NoAns F1 separately; a model that always abstains gets ~50 F1 overall.
  • Use `evaluate.load('squad_v2')` — it needs a `no_answer_probability` per prediction.
  • Few-shot prompting needs an explicit 'unanswerable' verbalisation; GPT-3 used the text 'unanswerable'.
08

References

All benchmarks