Commonsense Reasoning

Story Cloze

Story Cloze Test (ROCStories)

Tests whether a model can choose the right ending for a short five-sentence story about everyday life.

Released Apr 2016 Metric test accuracy (fine-tuned) SOTA 91.8 · Li et al. 2019 (2019)
01

At a glance

Task
Binary choice of story ending
Size
ROCStories: 98k five-sentence stories for training; Story Cloze: 1,871 validation + 1,871 test items (2016), 1,571 each (2018 release)
Metric
Accuracy
Source
Crowd-written everyday stories; wrong endings written by other workers
Access
Requires a form on the authors' site — not freely downloadable
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
Mostafazadeh et al. · NAACL 2016
Read the paper ↗ Hugging Face ↗
02

What it tests

ROCStories are tiny five-sentence stories about everyday life. The Story Cloze Test removes the last sentence and offers two candidates: the real ending and a plausible-sounding wrong one written by a different person.

Choosing correctly means tracking what the characters wanted and what would naturally follow — commonsense about causes and goals rather than facts.

GPT-1 fine-tuned on it (86.5%) and GPT-3 evaluated it few-shot (87.7%). It has mostly been retired since; the data needs a request form and the ending-writers left stylistic cues that models exploit.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Story
Karen was assigned a roommate her first year of college. Her roommate asked her to go to a nearby city for a concert. Karen agreed happily. The show was absolutely exhilarating.
Ending A
Karen became good friends with her roommate.
Ending B
Karen hated her roommate.
AnswerA
Story
Jim got his first credit card in college. He didn't have a job so he bought everything on his card. After he graduated he amounted a $10,000 debt. Jim realized that he was foolish to spend so much money.
Ending A
Jim decided to open another credit card.
Ending B
Jim decided to devise a plan for repayment.
AnswerB
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Fine-tuned (2016 test set)86.5 accPrior best 77.6 (val-LS-skip); an 8.9-point jump
Few-shot (70 examples), 2016 test set87.7 accZero-shot 83.2; fine-tuned SOTA then 91.8
05

State of the art

Best known · test accuracy (fine-tuned)
91.8
open · below human level
Model
Li et al. 2019 (as cited in GPT-3)
2019
Human baseline
~100

Few-shot: PaLM 540B 89.0 (5-shot). Rarely reported since 2022. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

input_sentence_1…4, sentence_quiz1, sentence_quiz2, answer_right_ending (1 or 2). Requires manual download: request the CSVs at cs.rochester.edu/nlp/rocstories and pass `data_dir=`.

python · load
from datasets import load_dataset

ds = load_dataset("story_cloze", "2016", split="test")
print(ds)            # columns: input_sentence_1…4, sentence_quiz1, sentence_quiz2, answer_right_ending (1 or 2)
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Multiple choice by length-normalised log-likelihood of each ending (acc_norm)
def choices(ex):
    if "endings" in ex: return ex["ctx"], ex["endings"], int(ex["label"])              # HellaSwag
    if "sol1" in ex:    return ex["goal"], [ex["sol1"], ex["sol2"]], ex["label"]       # PIQA
    story = " ".join(ex[f"input_sentence_{i}"] for i in range(1, 5))                  # Story Cloze
    return story, [ex["sentence_quiz1"], ex["sentence_quiz2"]], ex["answer_right_ending"] - 1
acc = 0
for ex in ds:
    ctx, ends, gold = choices(ex)
    s = [loglik(ctx, " " + e) / max(1, len(e)) for e in ends]
    acc += int(s.index(max(s)) == gold)
print("acc_norm", acc / len(ds))
4Or use lm-evaluation-harness

Task name(s): storycloze_2016. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks storycloze_2016 --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Not on the Hub as a free download — you must request access and point `load_dataset` at the local folder.
  • The 2016 and 2018 test sets differ in size; GPT-1/3 numbers are on 2016.
  • Ending-only baselines reach ~70% because wrong endings are stylistically different; report one.
08

References

All benchmarks