NLI

SNLI

Stanford Natural Language Inference

Tests whether a model can tell if one sentence logically follows from, contradicts, or is unrelated to another, using short sentences about everyday photos.

Released Sep 2015 Metric test accuracy SOTA 93.1 · EFL (2021)
01

At a glance

Task
3-way sentence-pair classification
Size
550k train / 10k dev / 10k test
Metric
Accuracy
Source text
Flickr30k image captions as premises; hypotheses written by Mechanical Turk workers
Licence
CC BY-SA 4.0
A large annotated corpus for learning natural language inference
Bowman, Angeli, Potts, Manning · EMNLP 2015
Read the paper ↗ Hugging Face ↗
02

What it tests

Natural language inference asks one question: if the first sentence is true, does the second one have to be true? There are three possible answers. It must be true (entailment), it cannot be true (contradiction), or we simply cannot tell (neutral).

SNLI was built by showing crowd-workers an image caption and asking them to write one sentence of each kind. Because the premise is always a caption of a real photo, the sentences are concrete and short, which keeps the task clean but also makes it easier than later datasets.

A model that genuinely understands the two sentences has to track who is doing what to whom, handle negation, and know a little about the world (a man sleeping is not inspecting a uniform). That is why NLI became the standard proxy for sentence understanding in the pre-LLM era.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Three labels for one premise
Premise
A man inspects the uniform of a figure in some East Asian country.
Hypothesis
The man is sleeping.
Answercontradiction
Premise
An older and younger man smiling.
Hypothesis
Two men are smiling and laughing at the cats playing on the floor.
Answerneutral
Premise
A soccer game with multiple males playing.
Hypothesis
Some men are playing a sport.
Answerentailment
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Fine-tuned (premise $ hypothesis concatenated with a delimiter)89.9 accPrior best 89.3 (CAFE ensemble)
05

State of the art

Best known · test accuracy
93.1
✓ saturated · at or above human level
Model
EFL (entailment as few-shot learner, RoBERTa-large)
2021
Human baseline
~88–89 (annotator agreement)

Leaderboard is informal and has been static since 2021; the benchmark is considered solved. Source: nlp.stanford.edu ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

premise, hypothesis, label (0 entailment · 1 neutral · 2 contradiction · −1 no gold label — filter these out)

python · load
from datasets import load_dataset

ds = load_dataset("stanfordnlp/snli", split="test")
print(ds)            # columns: premise, hypothesis, label (0 entailment · 1 neutral · 2 contradiction · −1 no gold label 
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Score each label by the log-likelihood of its verbaliser after the pair (GPT-3 style)
LABELS = ["True", "Neither", "False"]      # entailment / neutral / contradiction
def prompt(ex): return f"{ex['premise']}\nQuestion: {ex['hypothesis']} True, False, or Neither?\nAnswer:"
correct = 0
for ex in ds:
    if ex["label"] < 0: continue
    scores = [loglik(prompt(ex), " " + lab) for lab in LABELS]
    correct += int(scores.index(max(scores)) == ex["label"])
print("accuracy", correct / len(ds))
07

Pitfalls

  • About 2% of examples have label −1 (no annotator majority); drop them before scoring.
  • Hypothesis-only baselines reach ~67% accuracy because workers used predictable cues (negation words for contradiction, vague words for neutral). Report a hypothesis-only control if you care about reasoning.
  • Premises are image captions, so the domain is narrow; MultiNLI exists to fix that.
08

References

All benchmarks