Commonsense Reasoning

COPA

Choice of Plausible Alternatives

Tests causal common sense: given a short statement, which of two alternatives is the more likely cause or effect.

Released 2011 Metric SuperGLUE test accuracy SOTA 99.4 · Vega v2 (2022)
01

At a glance

Task
Binary choice; question is either 'cause' or 'effect'
Size
400 train / 100 dev / 500 test
Metric
Accuracy
Source
Hand-written by the authors, balanced so that both alternatives are plausible out of context
Part of
SuperGLUE
Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
Roemmele, Bejan, Gordon · AAAI Spring Symposium 2011
Read the paper ↗ Hugging Face ↗
02

What it tests

COPA is the smallest benchmark in this collection and one of the cleanest. A premise ('My body cast a shadow over the grass'), a direction ('what was the cause?'), and two alternatives ('the sun was rising' / 'the grass was cut'). Everything hinges on everyday causal knowledge.

It was written by hand in 2011 and designed so that neither alternative is implausible on its own — only the pairing with the premise decides.

GPT-3 reached 92% few-shot, and it was essentially solved (99.4%, human 100%) by 2022. It remains a useful smoke test because it is tiny and unambiguous.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

cause
Premise
My body cast a shadow over the grass.
Question
What was the CAUSE of this?
Alternatives
1. The sun was rising. · 2. The grass was cut.
Answer1
effect
Premise
The man broke his toe.
Question
What was the EFFECT of this?
Alternatives
1. He got a hole in his sock. · 2. He dropped a hammer on his foot.
Answer— trick: for 'effect' the answer is 1; for 'cause' it would be 2
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (32 examples), SuperGLUE92.0 accZero-shot 91.0; fine-tuned SOTA then 94.8
05

State of the art

Best known · SuperGLUE test accuracy
99.4
✓ saturated · at or above human level
Model
Vega v2
2022
Human baseline
100

ST-MoE 99.2, PaLM 540B 99.0. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

premise, choice1, choice2, question ('cause' / 'effect'), label (0 or 1)

python · load
from datasets import load_dataset

ds = load_dataset("aps/super_glue", "copa", split="validation")
print(ds)            # columns: premise, choice1, choice2, question ('cause' / 'effect'), label (0 or 1)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
def prompt(ex):
    conn = "because" if ex["question"] == "cause" else "so"
    return f"{ex['premise'][:-1]} {conn}"
def lower_first(s): return s[0].lower() + s[1:]
acc = 0
for ex in ds:
    s = [loglik(prompt(ex), " " + lower_first(ex[c])) for c in ("choice1", "choice2")]
    acc += int(s.index(max(s)) == ex["label"])
print("accuracy", acc / len(ds))
4Or use lm-evaluation-harness

Task name(s): copa. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks copa --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Connect premise and alternative with 'because' (cause) or 'so' (effect) when prompting an LM — GPT-3 did, and it matters.
  • 100 dev items: ±3 points is noise.
  • Lower-case the first letter of the alternative when appending it after the connective.
08

References

All benchmarks