Commonsense Reasoning

WinoGrande

WinoGrande

Tests whether a model can fill a blank in a sentence with the right of two people or objects, using common sense about the situation.

Released Jul 2019 Metric dev accuracy (XL, fine-tuned) SOTA 96.1 · ST-MoE-32B (2022)
01

At a glance

Task
Binary fill-in-the-blank (which of two options fills '_')
Size
40k train (sizes XS–XL), 1.3k dev, 1.8k test (hidden)
Metric
Accuracy
Construction
Crowd-workers write twin sentences; AfLite adversarial filtering removes items RoBERTa embeddings solve
Maintainer
Allen Institute for AI
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, Le Bras, Bhagavatula, Choi · AAAI 2020
Read the paper ↗ Hugging Face ↗
02

What it tests

WinoGrande takes the Winograd idea and makes it big enough to train on. Workers wrote pairs of sentences with a blank and two options, anchored on a random topic word to keep them diverse; then an algorithm (AfLite) removed every item that could be solved from shallow features.

The surviving items need the same kind of situational reasoning as WSC — 'The man could not lift his son because _ was so weak' — but there are tens of thousands, which also makes it a good training set for the original WSC.

GPT-3 scored 77.7% few-shot. Fine-tuned models have since passed the 94% human level, and it stayed in frontier model cards through 2024 (Claude 3 Opus 88.5% 5-shot).

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Sentence
The man could not lift his son because _ was so weak.
Options
1. the man · 2. the son
Answer1
Sentence
The man could not lift his son because _ was so heavy.
Options
1. the man · 2. the son
Answer2
Sentence
Sarah was a much better surgeon than Maria so _ always got the easier cases.
Options
1. Sarah · 2. Maria
Answer2
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (50 examples), partial scoring77.7 accZero-shot 70.2; fine-tuned RoBERTa 79.1; human 94.0
05

State of the art

Best known · dev accuracy (XL, fine-tuned)
96.1
✓ saturated · at or above human level
Model
ST-MoE-32B
2022
Human baseline
94.0

Few-shot best: Claude 3 Opus 88.5 (5-shot), GPT-4 87.5. UNICORN 91.3 on hidden test. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

sentence (contains '_'), option1, option2, answer ('1' or '2'). Configs winogrande_xs … winogrande_xl change only the train size.

python · load
from datasets import load_dataset

ds = load_dataset("allenai/winogrande", "winogrande_xl", split="validation")
print(ds)            # columns: sentence (contains '_'), option1, option2, answer ('1' or '2')
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Partial scoring: substitute each candidate into the sentence and score only the text *after* the blank
def split_on(ex):
    if "sentence" in ex:                      # WinoGrande: sentence with "_"
        pre, post = ex["sentence"].split("_"); opts = [ex["option1"], ex["option2"]]; gold = int(ex["answer"]) - 1
    else:                                     # SuperGLUE WSC: use span2 (pronoun) position
        t = ex["text"]; i = t.index(ex["span2_text"]); pre, post = t[:i], t[i + len(ex["span2_text"]):]
        opts = [ex["span1_text"]]; gold = ex["label"]
    return pre, post, opts, gold
acc = 0
for ex in ds:
    pre, post, opts, gold = split_on(ex)
    s = [loglik(pre + o, post) for o in opts]
    acc += int(s.index(max(s)) == gold)
print("accuracy", acc / len(ds))
4Or use lm-evaluation-harness

Task name(s): winogrande. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks winogrande --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Evaluate on validation; test labels are hidden behind the AI2 leaderboard.
  • Score by substituting each option into the blank and comparing the log-likelihood of the rest of the sentence (partial scoring).
  • Few-shot vs fine-tuned numbers are ~10 points apart; label which you mean.
08

References

All benchmarks