ReCoRD
Tests whether a model can fill a missing name or place in a sentence about a news article, which needs some common sense.
At a glance
What it tests
ReCoRD takes a news article and one of its own later sentences, masks an entity in that sentence, and asks which entity from the article belongs in the gap. The candidate set is every named entity in the passage.
Because the query sentence was written by a journalist as a continuation of the story, filling it correctly often needs an inference about what would plausibly happen next, not just entity matching — the authors filtered with models and humans to keep examples that need this.
GPT-3 did well here few-shot (90.2 acc), suggesting the task lines up with what language models learn naturally from news text.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Few-shot (32 examples), SuperGLUE | 90.2 acc / 91.1 F1 | Zero-shot 90.2 / 91.0; fine-tuned SOTA then 94.0 / 94.6 |
State of the art
Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
passage, query (contains '@placeholder'), entities (candidate list), answers (gold entity strings)
from datasets import load_dataset ds = load_dataset("aps/super_glue", "record", split="validation") print(ds) # columns: passage, query (contains '@placeholder'), entities (candidate list), answers (gold entity print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
acc = 0 for ex in ds: ctx = ex["passage"].replace("@highlight\n", "- ") + "\n" s = [loglik(ctx, ex["query"].replace("@placeholder", e)) for e in ex["entities"]] acc += int(ex["entities"][s.index(max(s))] in ex["answers"]) print("accuracy", acc / len(ds))
Task name(s): record. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks record --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Candidates are provided; score each by substituting it into the query and comparing log-likelihoods (what lm-eval does).
- The passage text contains '@highlight' bullet markers from the original CNN/DM format — keep or strip consistently.
- Multiple surface forms of the same entity can be gold; use the full answers list.
References
- Paper (arXiv 1810.12885)https://arxiv.org/abs/1810.12885
- Dataset pagehttps://sheng-z.github.io/ReCoRD-explorer/
- Hugging Face (super_glue/record)https://huggingface.co/datasets/aps/super_glue