QuAC
Tests whether a model can answer a student's questions about a Wikipedia article turn by turn, including saying when there is no answer.
At a glance
What it tests
QuAC is the information-seeking cousin of CoQA. A 'student' who cannot see the Wikipedia section asks questions; a 'teacher' who can see it answers with a span or says there is no answer. Because the student is guessing in the dark, questions are open-ended and often unanswerable.
The teacher also gives dialogue acts (whether to continue, whether to follow up), and the questions drift as the conversation develops, so context matters even more than in CoQA.
QuAC remains one of the few GPT-3-era benchmarks that is not solved: the best system is still about five F1 points below humans.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Few-shot (5 examples) | 44.3 F1 | Zero-shot 41.5; fine-tuned SOTA then 74.4 |
State of the art
Few-shot LLMs are far lower (Llama 3.1 405B 53.6, 1-shot). Source: quac.ai ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
context, questions, answers (texts, answer_starts), followups, yesnos. Test set hidden.
from datasets import load_dataset ds = load_dataset("allenai/quac", split="validation") print(ds) # columns: context, questions, answers (texts, answer_starts), followups, yesnos print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
# Feed the running dialogue; score each turn with token-F1 against the references def f1(pred, gold): p, g = pred.lower().split(), gold.lower().split(); common = sum(min(p.count(w), g.count(w)) for w in set(p)) if not common: return 0.0 pr, rc = common / len(p), common / len(g); return 2 * pr * rc / (pr + rc) scores = [] for ex in ds: hist = ex["story"] + "\n\n" for q, a in zip(ex["questions"], ex["answers"]["input_text"]): pred = generate(hist + f"Q: {q}\nA:", max_new_tokens=24, stop="\n").strip() scores.append(f1(pred, a)); hist += f"Q: {q}\nA: {a}\n" print("F1", sum(scores) / len(scores))
Pitfalls
- Predicting 'CANNOTANSWER' correctly is worth a lot; always include it as a candidate.
- Official F1 uses the max over human references; questions where humans disagree (F1 below a threshold) are filtered.
- Conversation history is essential — the first question alone rarely identifies the topic.
References
- Paper (arXiv 1808.07036)https://arxiv.org/abs/1808.07036
- Leaderboardhttps://quac.ai/
- Hugging Facehttps://huggingface.co/datasets/allenai/quac