Reading Comprehension

CoQA

Conversational Question Answering

Tests whether a model can answer a chain of questions about a passage, where each question depends on the earlier ones.

Released Aug 2018 Metric overall test F1 SOTA 90.7 · RoBERTa + AT + KD ensemble (2020)
01

At a glance

Task
Free-form answer + supporting span, in a dialogue
Size
8k conversations, 127k Q/A pairs; 7 domains (children's stories, literature, school exams, news, Wikipedia, Reddit, science)
Metric
Macro-averaged F1 against 4 reference answers (dev/test)
Answer types
Free-form text; also yes/no/unknown
Source
Two annotators chatting about a passage (questioner and answerer)
CoQA: A Conversational Question Answering Challenge
Reddy, Chen, Manning · TACL 2019
Read the paper ↗ Hugging Face ↗
02

What it tests

In CoQA two people talk about a passage. One asks questions, the other answers from the text. The questions refer back to earlier turns ('What did she do then?', 'Why?'), so answering turn 7 requires remembering turns 1–6.

Answers are free-form rather than exact spans, but each comes with a rationale span from the passage, and scoring uses token-level F1 against several human answers, so short, precise answers score best.

GPT-2 tried it zero-shot by appending 'A:' after the dialogue and reached 55 F1 — remarkable for a model never trained on QA. GPT-3 few-shot reached 85 F1, close to fine-tuned systems of the time.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Turns build on each other
Passage (excerpt)
Once upon a time, in a barn near a farm house, there lived a little white kitten named Cotton. Cotton lived high up in a nice warm place above the barn where all of the farmer's horses slept…
Q1
What color was Cotton?
A1
white
Q2
Where did she live?
A2
in a barn
Q3
Did she live alone?
A3
no
Q4
Who did she live with?
Answerwith her mommy and 5 sisters
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot: document + dialogue + 'A:' , greedy decoding55 F1 (dev)Matched or beat 3 of 4 supervised baselines; BERT SOTA then 89 F1
Few-shot (5 examples)85.0 F1Zero-shot 81.5; fine-tuned SOTA 90.7
05

State of the art

Best known · overall test F1
90.7
✓ saturated · at or above human level
Model
RoBERTa + AT + KD ensemble (Zhuiyi) / TR-MT (WeChat AI)
2020
Human baseline
88.8

Official leaderboard frozen since 2020. Source: stanfordnlp.github.io ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

story, questions (list), answers (dict with input_text, answer_start, answer_end). Test set is hidden; evaluate on validation.

python · load
from datasets import load_dataset

ds = load_dataset("stanfordnlp/coqa", split="validation")
print(ds)            # columns: story, questions (list), answers (dict with input_text, answer_start, answer_end)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Feed the running dialogue; score each turn with token-F1 against the references
def f1(pred, gold):
    p, g = pred.lower().split(), gold.lower().split(); common = sum(min(p.count(w), g.count(w)) for w in set(p))
    if not common: return 0.0
    pr, rc = common / len(p), common / len(g); return 2 * pr * rc / (pr + rc)
scores = []
for ex in ds:
    hist = ex["story"] + "\n\n"
    for q, a in zip(ex["questions"], ex["answers"]["input_text"]):
        pred = generate(hist + f"Q: {q}\nA:", max_new_tokens=24, stop="\n").strip()
        scores.append(f1(pred, a)); hist += f"Q: {q}\nA: {a}\n"
print("F1", sum(scores) / len(scores))
4Or use lm-evaluation-harness

Task name(s): coqa. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks coqa --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Feed the full dialogue history; evaluating turns independently drops F1 by 10+ points.
  • Use the official evaluation script (it strips articles/punctuation and takes max F1 over references).
  • Yes/no/unknown answers are ~20% of the data — make sure your decoding can produce them verbatim.
08

References

All benchmarks