Commonsense Reasoning

ARC

AI2 Reasoning Challenge (ARC-Easy / ARC-Challenge)

Tests whether a model can answer grade-school science exam questions; the Challenge split contains the ones simple word-matching cannot solve.

Released Mar 2018 Metric accuracy (0-shot / 25-shot) SOTA 96.9 (Challenge) · 98.9 (Easy) · Llama 3.1 405B (2024)
01

At a glance

Task
Multiple choice (usually 4 options)
Size
ARC-Easy 5.2k / ARC-Challenge 2.6k questions (train/dev/test each)
Metric
Accuracy
Source
US grade 3–9 standardised science exams; comes with a 14M-sentence science corpus
Maintainer
Allen Institute for AI
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, Cowhey, Etzioni, Khot, Sabharwal, Schoenick, Tafjord · arXiv 2018
Read the paper ↗ Hugging Face ↗
02

What it tests

ARC is a pile of real science-exam questions. The authors ran two simple solvers over all of them — one based on retrieval, one on word co-occurrence — and put every question both solvers failed into the Challenge set. Everything else is Easy.

So ARC-Challenge is defined by what does not work: you cannot answer 'Which property of a mineral can be determined just by looking at it?' by matching keywords to a textbook sentence. You need to combine facts or reason about the options.

GPT-3 reached 51.5% on Challenge few-shot (and 70.1% on Easy). Frontier models now sit at 96–97% on Challenge, so it has effectively been retired in favour of harder sets like GPQA.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

ARC-Challenge
Question
Which property of a mineral can be determined just by looking at it?
Options
A. luster · B. mass · C. weight · D. hardness
AnswerA
ARC-Easy
Question
Which of the following is a renewable resource?
Options
A. coal · B. oil · C. natural gas · D. wind
AnswerD
ARC-Challenge
Question
George wants to warm his hands quickly by rubbing them. Which skin surface will produce the most heat?
Options
A. dry palms · B. wet palms · C. palms covered with oil · D. palms covered with lotion
AnswerA
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (50 examples)Challenge 51.5Easy 70.1Zero-shot 51.4 / 68.8; fine-tuned SOTA then 78.5 (UnifiedQA) / 92.0
05

State of the art

Best known · accuracy (0-shot / 25-shot)
96.9 (Challenge) · 98.9 (Easy)
✓ effectively saturated
Model
Llama 3.1 405B (Challenge) · DeepSeek-V3 base (Easy)
2024
Human baseline
not published; near ceiling

GPT-4o and Claude 3.5 Sonnet 96.7 on Challenge. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

question, choices (dict with text list and label list), answerKey ('A'–'E' or '1'–'4')

python · load
from datasets import load_dataset

ds = load_dataset("allenai/ai2_arc", "ARC-Challenge", split="test")
print(ds)            # columns: question, choices (dict with text list and label list), answerKey ('A'–'E' or '1'–'4')
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Science MC: score each option text as a continuation of "Question: … Answer:"
acc = 0
for ex in ds:
    q = ex.get("question") or ex["question_stem"]
    texts, labels = ex["choices"]["text"], ex["choices"]["label"]
    s = [loglik(f"Question: {q}\nAnswer:", " " + t) / max(1, len(t)) for t in texts]
    acc += int(labels[s.index(max(s))] == ex["answerKey"])
print("acc_norm", acc / len(ds))
4Or use lm-evaluation-harness

Task name(s): arc_challenge, arc_easy. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks arc_challenge --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Some questions have 3 or 5 options, and some use numeric labels — handle both.
  • lm-eval's arc_challenge scores the option *text* by log-likelihood (acc_norm); the 'MMLU-style' letter prompt used by model cards gives much higher numbers. Say which.
  • Easy and Challenge are reported separately; never average them.
08

References

All benchmarks