MRPC
Tests whether a model can tell if two news sentences say the same thing in different words.
At a glance
What it tests
Two sentences from different news articles about the same event: do they say the same thing? That is MRPC. Because both sentences cover the same story, they share most of their content words, so the model has to notice the differences rather than the overlap.
The dataset is small and two-thirds of the pairs are paraphrases, which is why F1 is reported alongside accuracy — always predicting 'paraphrase' already gets 68% accuracy.
GPT-1 reached 82.3 F1, a modest gain over the prior best; large pre-trained models have since passed the 86.3 F1 human baseline by a wide margin.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned; both sentence orderings fed and summed | 82.3 F1 | Prior best 86.0 (single-task BiLSTM+ELMo+Attn had 84.4 on GLUE) |
State of the art
Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
sentence1, sentence2, label (0 not_equivalent · 1 equivalent)
from datasets import load_dataset ds = load_dataset("nyu-mll/glue", "mrpc", split="validation") print(ds) # columns: sentence1, sentence2, label (0 not_equivalent · 1 equivalent) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
# Binary pair classification: compare log P(" yes") vs log P(" no") def prompt(ex): k = [c for c in ex if c not in ("label", "idx", "word")][:2] return f"Sentence 1: {ex[k[0]]}\nSentence 2: {ex[k[1]]}\nQuestion: Do these mean the same thing? yes or no?\nAnswer:" acc = sum(int((loglik(prompt(ex), " yes") > loglik(prompt(ex), " no")) == bool(ex["label"])) for ex in ds) / len(ds) print("accuracy", acc)
Task name(s): mrpc. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks mrpc --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- 408 dev examples — variance between seeds is 1–2 F1.
- Report F1 of the positive class, not macro-F1, to match GLUE.
- Order of the two sentences should not matter; GPT-1 averaged over both orders.
References
- Paper (IWP 2005)https://aclanthology.org/I05-5002/
- GLUEhttps://gluebenchmark.com/
- Hugging Face (GLUE/mrpc)https://huggingface.co/datasets/nyu-mll/glue