Commonsense Reasoning

Winograd Schema Challenge

Winograd Schema Challenge (WSC273 / SuperGLUE WSC)

Tests whether a model can work out who a pronoun refers to in sentences where only common sense decides — 'it' in 'the trophy does not fit in the suitcase because it is too big'.

Released 2012 Metric accuracy SOTA ~91.6 (WSC273) · OK-Transformer (2023)
01

At a glance

Task
Resolve an ambiguous pronoun to one of two noun phrases
Size
WSC273: 273 items (no training set). SuperGLUE WSC: 554 train / 104 dev / 146 test
Metric
Accuracy
Construction
Expert-written sentence pairs that differ in one word, each flipping the correct referent
Part of
SuperGLUE (as WSC)
The Winograd Schema Challenge
Levesque, Davis, Morgenstern · KR 2012
Read the paper ↗ Hugging Face ↗
02

What it tests

Terry Winograd's example from 1972 is the whole idea: 'The city councilmen refused the demonstrators a permit because they feared violence' — who feared violence? Swap 'feared' for 'advocated' and the answer swaps too. Nothing about grammar changes; only knowledge of how councils and demonstrators behave does.

Because the sentence pairs are minimal, statistical tricks (which noun is usually the subject, which word co-occurs with 'violence') fail — that was the design goal when it was proposed as an alternative to the Turing test.

GPT-2 solved it by scoring the sentence with each candidate substituted in and picking the more probable one, reaching 70.7% zero-shot; GPT-3 reached 88.6% few-shot. Human agreement is in the low-to-mid 90s.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

One word flips the answer
Sentence
The trophy doesn't fit in the brown suitcase because it is too big.
Question
What is too big?
Answerthe trophy
Sentence
The trophy doesn't fit in the brown suitcase because it is too small.
Question
What is too small?
Answerthe suitcase
Sentence
The city councilmen refused the demonstrators a permit because they feared violence.
Question
Who feared violence?
Answerthe city councilmen
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot, partial scoring (Trinh & Le 2018) on WSC27370.70 accPrior best 63.7 (Trinh & Le ensemble)
Few-shot (70 examples) on WSC273; also SuperGLUE WSC few-shot88.6 (WSC273)80.1 (SuperGLUE WSC)Zero-shot WSC273 88.3; fine-tuned SOTA 90.1
05

State of the art

Best known · accuracy
~91.6 (WSC273)
open · below human level
Model
OK-Transformer (RoBERTa-based) / RoBERTa fine-tuned on WinoGrande 90.1
2023
Human baseline
92–95

SuperGLUE WSC: 97.3 (Vega v2) vs human 100. Kocijan et al. (2022) argue WSC273 is 'defeated' by large LMs. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

SuperGLUE: text, span1_text, span2_text, label (0/1). WSC273: `load_dataset('winograd_wsc', 'wsc273')` with text, pronoun, options, label.

python · load
from datasets import load_dataset

ds = load_dataset("aps/super_glue", "wsc.fixed", split="validation")
print(ds)            # columns: SuperGLUE: text, span1_text, span2_text, label (0/1)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Partial scoring: substitute each candidate into the sentence and score only the text *after* the blank
def split_on(ex):
    if "sentence" in ex:                      # WinoGrande: sentence with "_"
        pre, post = ex["sentence"].split("_"); opts = [ex["option1"], ex["option2"]]; gold = int(ex["answer"]) - 1
    else:                                     # SuperGLUE WSC: use span2 (pronoun) position
        t = ex["text"]; i = t.index(ex["span2_text"]); pre, post = t[:i], t[i + len(ex["span2_text"]):]
        opts = [ex["span1_text"]]; gold = ex["label"]
    return pre, post, opts, gold
acc = 0
for ex in ds:
    pre, post, opts, gold = split_on(ex)
    s = [loglik(pre + o, post) for o in opts]
    acc += int(s.index(max(s)) == gold)
print("accuracy", acc / len(ds))
4Or use lm-evaluation-harness

Task name(s): wsc273 (original) · wsc (SuperGLUE). The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks wsc273 --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Use the 'wsc.fixed' config — the original SuperGLUE release had span offset bugs.
  • 273 items means one item is 0.37 points; differences under 2 points are noise.
  • Partial scoring (only score tokens after the pronoun) is what makes LM-based evaluation work; full-sentence scoring is worse.
08

References

All benchmarks