Commonsense Reasoning

PIQA

Physical Interaction: Question Answering

Tests physical common sense: given an everyday goal, which of two near-identical instructions actually works in the real world.

Released Nov 2019 Metric test accuracy (fine-tuned) SOTA 90.1 · UNICORN (2021)
01

At a glance

Task
Binary choice of solution for a goal
Size
16k train / 1.8k dev / 3k test (hidden)
Metric
Accuracy
Source
Goals from instructables.com; wrong solutions are minimal edits of the right one written by annotators
Licence
AFL 3.0
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Zellers, Le Bras, Gao, Choi · AAAI 2020
Read the paper ↗ Hugging Face ↗
02

What it tests

PIQA tests physical commonsense — the kind of knowledge people pick up by handling objects and almost never write down. Each item is a goal ('Make an outdoor pillow') and two solutions that differ by a small edit; one works and one does not ('…fill with cotton' vs '…fill with ice').

Because the wrong answer reuses most of the right one, a model cannot rely on topic words; it has to know that ice melts, that foil conducts heat, that you need a hole to thread a string.

GPT-3 reached 82.8% few-shot (zero-shot 81.0 was already above the fine-tuned RoBERTa of the time at 79.4). The human baseline of 94.9% has still not been reached in the few-shot setting.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Goal
To separate egg whites from the yolk using a water bottle, you should…
Solution 1
Squeeze the water bottle and press it against the yolk. Release, which creates suction and lifts the yolk.
Solution 2
Place the water bottle and press it against the yolk. Keep pushing, which creates suction and lifts the yolk.
AnswerSolution 1
Goal
How do I ready a guinea pig cage for its new occupants?
Solution 1
Provide the guinea pig with a cage full of a few inches of bedding made of ripped paper strips, you will also need to supply it with a water bottle and a food dish.
Solution 2
Provide the guinea pig with a cage full of a few inches of bedding made of ripped jeans material, you will also need to supply it with a water bottle and a food dish.
AnswerSolution 1
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (50 examples)82.8 accZero-shot 81.0, one-shot 80.5; fine-tuned SOTA then 79.4 (RoBERTa). Flagged for possible test-set contamination in the paper's analysis
05

State of the art

Best known · test accuracy (fine-tuned)
90.1
open · below human level
Model
UNICORN (T5-11B multitask)
2021
Human baseline
94.9

Few-shot: Llama 3.1 405B 85.9 (0-shot), PaLM 540B 85.2. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

goal, sol1, sol2, label (0 or 1). Test labels hidden.

python · load
from datasets import load_dataset

ds = load_dataset("ybisk/piqa", split="validation")
print(ds)            # columns: goal, sol1, sol2, label (0 or 1)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Multiple choice by length-normalised log-likelihood of each ending (acc_norm)
def choices(ex):
    if "endings" in ex: return ex["ctx"], ex["endings"], int(ex["label"])              # HellaSwag
    if "sol1" in ex:    return ex["goal"], [ex["sol1"], ex["sol2"]], ex["label"]       # PIQA
    story = " ".join(ex[f"input_sentence_{i}"] for i in range(1, 5))                  # Story Cloze
    return story, [ex["sentence_quiz1"], ex["sentence_quiz2"]], ex["answer_right_ending"] - 1
acc = 0
for ex in ds:
    ctx, ends, gold = choices(ex)
    s = [loglik(ctx, " " + e) / max(1, len(e)) for e in ends]
    acc += int(s.index(max(s)) == gold)
print("acc_norm", acc / len(ds))
4Or use lm-evaluation-harness

Task name(s): piqa. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks piqa --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • GPT-3 flagged PIQA as a contamination risk — their overlap-cleaned score was ~3 points lower.
  • Score 'goal + solution' as a sequence and compare log-likelihoods; the two solutions are nearly the same length so normalisation matters little.
  • Dev set is small (1.8k); report confidence intervals.
08

References

All benchmarks