QNLI
Tests whether a model can tell if a given sentence from Wikipedia actually contains the answer to a question.
At a glance
What it tests
QNLI was manufactured by the GLUE authors from SQuAD. Every SQuAD question is paired with each sentence of its paragraph; the pair is labelled entailment if that sentence contains the answer, otherwise not entailment.
That turns an extractive QA dataset into a sentence-pair problem, which is a format every NLI model can consume. It rewards the same skill as retrieval: deciding whether a passage is relevant to a question, without having to produce the answer span.
It is one of the easier GLUE tasks for pre-trained transformers because lexical overlap between question and sentence is a very strong signal.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned | 88.1 acc | Prior best 82.3 (CAFE) |
State of the art
Saturated well beyond the human baseline. Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
question, sentence, label (0 entailment · 1 not_entailment)
from datasets import load_dataset ds = load_dataset("nyu-mll/glue", "qnli", split="validation") print(ds) # columns: question, sentence, label (0 entailment · 1 not_entailment) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
LABELS = ["True", "False"] # entailment / not_entailment def prompt(ex): a = ex.get("premise") or ex.get("question"); b = ex.get("hypothesis") or ex.get("sentence") return f"{a}\nQuestion: {b} True or False?\nAnswer:" acc = sum(int(max(range(2), key=lambda i: loglik(prompt(ex), " " + LABELS[i])) == ex["label"]) for ex in ds) / len(ds) print("accuracy", acc)
Task name(s): qnli. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks qnli --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Label 0 means entailment (the opposite of what the name 'not_entailment=1' suggests to some); check `features['label'].names`.
- Because pairs come from the same paragraph, negatives are topically related, not random; keep that in mind when using QNLI as a retrieval proxy.
References
- GLUE paperhttps://arxiv.org/abs/1804.07461
- GLUE sitehttps://gluebenchmark.com/
- Hugging Face (GLUE/qnli)https://huggingface.co/datasets/nyu-mll/glue