Synthetic / Probing

GPT-3 synthetic tasks

Arithmetic, word scrambling, SAT analogies, news generation, novel words, grammar correction

A set of made-up probe tasks — multi-digit arithmetic, unscrambling letters, SAT-style word analogies and spotting machine-written news — used to test learning from examples in the prompt.

Released May 2020 Metric no shared leaderboard SOTA n/a · — (2020)
01

At a glance

Arithmetic
10 sub-tasks: 2/3/4/5-digit addition and subtraction, 2-digit multiplication, 1-digit composite; 2,000 random problems each
Word manipulation
Cycle letters, anagrams (first/last letter fixed), random insertion, reversed words; 10k words each
SAT analogies
374 five-way analogy questions from SAT exams before 2005
News generation
Human raters judge whether ~200-word articles are model- or human-written
Also
Using a novel word in a sentence; correcting English grammar (qualitative only)
Language Models are Few-Shot Learners (Section 3.9)
Brown et al. (OpenAI) · NeurIPS 2020
Read the paper ↗
02

What it tests

Alongside the public benchmarks, the GPT-3 paper built a handful of tasks from scratch to ask a narrower question: can the model learn a *new* skill from the examples in its prompt, rather than recall something from pre-training?

Arithmetic is the cleanest case. The 175B model got 100% on 2-digit addition and 98.9% on 2-digit subtraction few-shot, but fell to ~10% on 5-digit problems and 29% on 2-digit multiplication; smaller models were near zero. The authors checked for memorisation and found fewer than 0.8% of problems in the training data.

Word scrambling tests a similar thing with letters: 'inoffensive' from 'noffensive i'? The model managed reversed words only 0.4% of the time — a direct symptom of BPE tokenisation, which hides individual letters. SAT analogies (65% vs ~57% for college applicants) and the news-article Turing test (humans at 52% accuracy, barely above chance) rounded out the section.

None of these are downloadable benchmarks, but they shaped later evaluation design: GSM8K-style arithmetic, letter-level manipulation tasks in BIG-bench, and human-preference studies all descend from here.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Arithmetic (few-shot)
Prompt
Q: What is 48 plus 76?\nA: 124\nQ: What is 97 minus 29?\nA:
Answer68 — 175B few-shot: 2D+ 100%, 2D− 98.9%, 3D+ 80.4%, 4D+ 25.5%, 5D+ 9.3%, 2Dx 29.2%
Cycle letters
Prompt
Please unscramble the letters into a word, and write that word:\nlyinevitab =
Answerinevitably — 175B few-shot 37.9%
SAT analogy
Prompt
audacious is to boldness as (a) sanctimonious is to hypocrisy, (b) anonymous is to identity, (c) remorseful is to misdeed, (d) deleterious is to result, (e) impressionable is to temptation
Answer(a) — 175B few-shot 65.2%
News generation
Setup
Humans shown a title + subtitle, then a ~200-word article from a model or a human
AnswerMean human accuracy at spotting the 175B model's articles: 52% (chance 50%)
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (50 examples) unless stated2-digit addition 100%3-digit addition 80.4%cycle letters 37.9%anagrams-1 15.1%reversed 0.4%SAT analogies 65.2%news detection 52%All on the 175B model; results scale smoothly with size
05

State of the art

Best known · no shared leaderboard
n/a
✓ saturated · at or above human level
Model
—
2020
Human baseline
SAT analogies ~57 (college applicants); news detection 52% means indistinguishable

Modern models solve multi-digit arithmetic with chain-of-thought or tools; letter manipulation remains tokeniser-limited. The successors are GSM8K, BIG-bench and human-preference evals. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

lm-eval ships regenerated versions of the arithmetic tasks (`EleutherAI/arithmetic` on the Hub). Word-scrambling and analogy sets are not published; generate your own with the recipe in Section 3.9.

python · load
from datasets import load_dataset
ds = load_dataset("EleutherAI/arithmetic", "arithmetic_2da", split="validation")   # 2-digit addition, regenerated by EleutherAI
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Regenerate GPT-3's arithmetic probe: 2,000 random problems per sub-task, exact-match scoring
import random
def problems(digits, op, n=2000, seed=0):
    r = random.Random(seed); lo, hi = 10 ** (digits - 1), 10 ** digits - 1
    for _ in range(n):
        a, b = r.randint(lo, hi), r.randint(lo, hi)
        yield f"Q: What is {a} {'plus' if op == '+' else 'minus'} {b}?\nA:", str(a + b if op == '+' else a - b)
FEW = "Q: What is 48 plus 76?\nA: 124\n\nQ: What is 97 minus 29?\nA: 68\n\n"
for d in (2, 3, 4, 5):
    for op in "+-":
        ps = list(problems(d, op)); ok = sum(generate(FEW + q, max_new_tokens=8, stop="\n").strip() == a for q, a in ps)
        print(f"{d}-digit {op}: {ok / len(ps):.1%}")
4Or use lm-evaluation-harness

Task name(s): arithmetic_2da, arithmetic_2ds, arithmetic_3da, …, arithmetic_5ds, arithmetic_2dm, arithmetic_1dc. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks arithmetic_2da --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Results depend heavily on the number format (commas, spaces) and the tokeniser; compare models only under the same formatting.
  • Arithmetic 'accuracy' here is exact string match of the integer answer.
  • The news-detection study used 80 US participants — a small sample.
08

References

All benchmarks