NLI

ANLI

Adversarial NLI

Tests whether a model can judge if one sentence follows from another, using examples deliberately written to trick strong models.

Released Oct 2019 Metric R3 test accuracy (fine-tuned) SOTA 74.7 · ST-MoE-32B (2022)
01

At a glance

Task
3-way NLI
Size
R1 17k / R2 45k / R3 100k train; 1k dev and 1k test per round (R3 1.2k)
Metric
Accuracy per round (R1, R2, R3) and averaged
Construction
Human-and-model-in-the-loop: writers craft hypotheses until a BERT (R1), RoBERTa (R2) or RoBERTa-ensemble (R3) model gets them wrong
Maintainer
Facebook AI Research
Adversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Williams, Dinan, Bansal, Weston, Kiela · ACL 2020
Read the paper ↗ Hugging Face ↗
02

What it tests

ANLI turned dataset construction into an adversarial game. A writer is shown a premise and a target label and asked to write a hypothesis that a strong model will get wrong. Examples that fool the model (and that other humans verify) are kept.

Each round uses a stronger adversary trained on the previous rounds, so R3 contains the examples that beat an ensemble of RoBERTa models trained on R1+R2+SNLI+MNLI. The premises in R3 are also longer and come from more varied sources.

This is why ANLI stayed hard long after SNLI and MNLI saturated. GPT-3 at 175B scored only ~40% on R3 few-shot, near chance, and the paper flagged it as one of the tasks where in-context learning clearly underperformed fine-tuning.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Round 1 (BERT as adversary)
Premise
The Parma trolleybus system (Italian: Rete filoviaria di Parma) forms part of the public transport network of the city and comune of Parma, in the region of Emilia-Romagna, northern Italy. In operation since 1953, the system presently comprises four urban routes.
Hypothesis
The trolleybus system has over 2 urban routes
Answerentailment
Round 3 (RoBERTa ensemble as adversary)
Premise
Kevin Joseph Aloysius 'Chuck' Connors (April 10, 1921 – November 10, 1992) was an American actor, writer and professional basketball and baseball player. He is one of only 13 athletes in the history of American professional sports to have played in both Major League Baseball and the National Basketball Association.
Hypothesis
Chuck Connors was born in a leap year.
Answerneutral (1921 is not a leap year, but the premise does not say so — the model must know it)
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (50 examples)R1 36.8R2 34.0R3 40.2Zero-shot R3 40.2; fine-tuned RoBERTa-large then 54.0 on R3
05

State of the art

Best known · R3 test accuracy (fine-tuned)
74.7
no human baseline
Model
ST-MoE-32B
2022
Human baseline
not defined (adversarially built)

Few-shot frontier models still trail fine-tuned encoders here; PaLM 540B 5-shot R3 was 51.2. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

premise, hypothesis, label (0 entailment · 1 neutral · 2 contradiction), reason (free-text annotator explanation)

python · load
from datasets import load_dataset

ds = load_dataset("facebook/anli", split="test_r3")
print(ds)            # columns: premise, hypothesis, label (0 entailment · 1 neutral · 2 contradiction), reason (free-text
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Score each label by the log-likelihood of its verbaliser after the pair (GPT-3 style)
LABELS = ["True", "Neither", "False"]      # entailment / neutral / contradiction
def prompt(ex): return f"{ex['premise']}\nQuestion: {ex['hypothesis']} True, False, or Neither?\nAnswer:"
correct = 0
for ex in ds:
    if ex["label"] < 0: continue
    scores = [loglik(prompt(ex), " " + lab) for lab in LABELS]
    correct += int(scores.index(max(scores)) == ex["label"])
print("accuracy", correct / len(ds))
4Or use lm-evaluation-harness

Task name(s): anli_r1, anli_r2, anli_r3. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks anli_r1 --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Report all three rounds; R1 alone is far easier than R3.
  • The 'reason' field explains why the example fooled the model — useful for error analysis, but never feed it to the model being tested.
  • Few-shot prompting is sensitive to how the three labels are verbalised ('True / Neither / False' is the GPT-3 convention).
08

References

All benchmarks