DROP
Tests whether a model can read a paragraph and do simple arithmetic, counting or date comparison to answer a question.
At a glance
What it tests
DROP was built to break span-extraction models. Workers wrote questions while a BiDAF model watched; a question was accepted only if BiDAF got it wrong. The result is questions like 'How many more yards was the longest touchdown than the shortest?' whose answer is not written anywhere in the text.
Answering requires reading the right numbers, then doing something with them — subtracting, counting, comparing dates, or collecting several spans. That is why DROP became the standard test of 'numerical reasoning' and why GPT-3 did poorly on it few-shot (36.5 F1).
Modern chat models with chain-of-thought now reach the low 90s, but the human baseline of 96.4 F1 has not been passed.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Few-shot (20 examples) | 36.5 F1 | Zero-shot 23.6; fine-tuned SOTA then 89.1 (QDGAT); even BERT-based baselines beat GPT-3 |
State of the art
Llama 3.1 405B 88.7, Claude 3.5 Sonnet 88.3, GPT-4o 83.7 in the same table. Source: huggingface.co ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
passage, question, answers_spans (spans, types: 'number' / 'date' / 'span')
from datasets import load_dataset ds = load_dataset("ucinlp/drop", split="validation") print(ds) # columns: passage, question, answers_spans (spans, types: 'number' / 'date' / 'span') print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
import evaluate metric = evaluate.load("squad_v2") # for DROP use the official drop_eval.py from allenai/allennlp-reading-comprehension preds, refs = [], [] for ex in ds: p = f"{ex['context'] if 'context' in ex else ex['passage']}\nQuestion: {ex['question']}\nAnswer (or 'unanswerable'):" out = generate(p, max_new_tokens=32, stop="\n").strip() preds.append({"id": ex["id"], "prediction_text": "" if out.lower().startswith("unanswerable") else out, "no_answer_probability": float(out.lower().startswith("unanswerable"))}) refs.append({"id": ex["id"], "answers": ex["answers"]}) print(metric.compute(predictions=preds, references=refs))
Task name(s): drop. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks drop --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Use the official DROP scorer: numbers are compared after normalisation and multi-span answers are matched as bags.
- Few-shot results depend heavily on whether the prompt allows chain-of-thought; say so.
- Some questions have multiple valid answers; F1 takes the max over them.
References
- Paper (arXiv 1903.00161)https://arxiv.org/abs/1903.00161
- AI2 leaderboardhttps://leaderboard.allenai.org/drop/
- Hugging Facehttps://huggingface.co/datasets/ucinlp/drop