Commonsense Reasoning

HellaSwag

HellaSwag (Harder Endings, Longer contexts, Low-shot Activities for Situations With Adversarial Generations)

Tests whether a model can pick the most sensible continuation of an everyday scene from four options written to fool it.

Released May 2019 Metric accuracy (10-shot) SOTA 95.4 · Claude 3 Opus (2024)
01

At a glance

Task
4-way multiple choice sentence completion
Size
40k train / 10k dev / 10k test
Metric
Accuracy
Source
ActivityNet video captions and WikiHow articles; wrong endings generated by a language model and filtered adversarially (AF) so BERT cannot tell them apart
Predecessor
SWAG (2018)
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, Holtzman, Bisk, Farhadi, Choi · ACL 2019
Read the paper ↗ Hugging Face ↗
02

What it tests

HellaSwag gives a short description of an everyday situation ('A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. She…') and four ways the sentence could continue. One is the real continuation; three were written by a model.

The trick is adversarial filtering: generated endings that BERT could identify as fake were thrown away and regenerated, until what remained were endings that read naturally to a model but are obviously wrong to a person (humans score ~95%).

GPT-3 reached 79.3% few-shot, and the benchmark stayed in every model report from then until it saturated around 2024 (Claude 3 Opus 95.4%, GPT-4 95.3%).

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Context
A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. She…
Options
A. rinses the bucket off with soap and blow dries the dog's head. · B. uses a hose to keep it from getting soapy. · C. gets the dog wet, then it runs away again. · D. gets into a bath tub with the dog.
AnswerC
Context
Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles. then…
Options
A. , the man adds wax to the windshield and cuts it. · B. , a person board a ski lift, while two men supporting the head of the person wearing winter clothes snow as the we girls sled. · C. , the man puts on a christmas coat, knitted with netting. · D. , the man continues removing the snow on his car.
AnswerD
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (20 examples), log-likelihood of each ending79.3 accZero-shot 78.9; fine-tuned SOTA then 85.6 (ALUM)
05

State of the art

Best known · accuracy (10-shot)
95.4
✓ saturated · at or above human level
Model
Claude 3 Opus
2024
Human baseline
95.6

GPT-4 95.3 (10-shot). Frontier models stopped reporting it in 2025. Source: www-cdn.anthropic.com ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

ctx, endings (list of 4), label (string '0'–'3'), activity_label. Test labels hidden; use validation.

python · load
from datasets import load_dataset

ds = load_dataset("Rowan/hellaswag", split="validation")
print(ds)            # columns: ctx, endings (list of 4), label (string '0'–'3'), activity_label
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Multiple choice by length-normalised log-likelihood of each ending (acc_norm)
def choices(ex):
    if "endings" in ex: return ex["ctx"], ex["endings"], int(ex["label"])              # HellaSwag
    if "sol1" in ex:    return ex["goal"], [ex["sol1"], ex["sol2"]], ex["label"]       # PIQA
    story = " ".join(ex[f"input_sentence_{i}"] for i in range(1, 5))                  # Story Cloze
    return story, [ex["sentence_quiz1"], ex["sentence_quiz2"]], ex["answer_right_ending"] - 1
acc = 0
for ex in ds:
    ctx, ends, gold = choices(ex)
    s = [loglik(ctx, " " + e) / max(1, len(e)) for e in ends]
    acc += int(s.index(max(s)) == gold)
print("acc_norm", acc / len(ds))
4Or use lm-evaluation-harness

Task name(s): hellaswag. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks hellaswag --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Score endings by length-normalised log-likelihood (acc_norm); raw log-likelihood favours short endings.
  • The text contains artefacts like '[header]' and '[title]' from WikiHow; lm-eval strips them with a preprocess function.
  • Label is a string in the HF version; cast to int.
08

References

All benchmarks