MultiNLI
Tests whether a model can judge if one sentence follows from another across many kinds of writing — fiction, phone calls, government reports and more.
At a glance
What it tests
MultiNLI keeps the SNLI recipe (premise, hypothesis, three labels) but draws premises from ten very different kinds of text, from telephone transcripts to government reports. The point is to test whether a model's understanding transfers across writing styles rather than memorising caption-speak.
The test set is split in two. The matched half uses the same genres as training. The mismatched half uses five genres the model has never seen, so the gap between the two numbers is a direct measure of over-fitting to style.
Because it is large, varied and part of GLUE, MNLI became the default dataset for pre-training-then-fine-tuning papers from BERT onward. A GPT-1 result on MNLI is a result on exactly this data.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned | 82.1 (matched) / 81.4 (mismatched) | Prior best 80.6 / 80.1 (GenSen / multi-task) |
State of the art
GLUE leaderboard snapshot reproduced in the Vega paper; the leaderboard itself has been frozen since 2022. Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
premise, hypothesis, label (0 entailment · 1 neutral · 2 contradiction). Test labels are hidden; use validation_matched / validation_mismatched.
from datasets import load_dataset ds = load_dataset("nyu-mll/glue", "mnli", split="validation_matched") print(ds) # columns: premise, hypothesis, label (0 entailment · 1 neutral · 2 contradiction) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
# Score each label by the log-likelihood of its verbaliser after the pair (GPT-3 style) LABELS = ["True", "Neither", "False"] # entailment / neutral / contradiction def prompt(ex): return f"{ex['premise']}\nQuestion: {ex['hypothesis']} True, False, or Neither?\nAnswer:" correct = 0 for ex in ds: if ex["label"] < 0: continue scores = [loglik(prompt(ex), " " + lab) for lab in LABELS] correct += int(scores.index(max(scores)) == ex["label"]) print("accuracy", correct / len(ds))
Task name(s): mnli. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks mnli --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Test labels are only available through the GLUE server; most papers now report dev accuracy. Say which you used.
- Always report both matched and mismatched; a large gap means the model learned genre cues.
- Hypothesis-only artefacts exist here too (~53% without the premise).
References
- Paper (arXiv 1704.05426)https://arxiv.org/abs/1704.05426
- Dataset pagehttps://cims.nyu.edu/~sbowman/multinli/
- Hugging Face (GLUE/mnli)https://huggingface.co/datasets/nyu-mll/glue