SciTail
Tests whether a model can decide if a sentence found on the web supports a statement taken from a school science exam.
At a glance
What it tests
SciTail is NLI in the wild. Instead of asking workers to write hypotheses, the authors took multiple-choice science questions, turned each question–answer pair into a statement, and paired it with sentences a search engine returned for that question.
The premise therefore was not written to match the hypothesis, so the two sentences often share vocabulary without one implying the other. That removes the annotation artefacts of SNLI and makes the dataset harder than its size suggests.
GPT-1 used it as its fourth NLI dataset and gained five points over the previous best, one of the clearer signs that generative pre-training transferred to a new domain.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned | 88.3 acc | Prior best 83.3 (CAFE) |
State of the art
Official AI2 leaderboard; CA-MTL and ALUM tied at 96.75. Source: leaderboard.allenai.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
premise, hypothesis, label (string: 'entails' / 'neutral')
from datasets import load_dataset ds = load_dataset("allenai/scitail", "tsv_format", split="test") print(ds) # columns: premise, hypothesis, label (string: 'entails' / 'neutral') print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
LABELS = ["True", "False"] # entailment / not_entailment def prompt(ex): a = ex.get("premise") or ex.get("question"); b = ex.get("hypothesis") or ex.get("sentence") return f"{a}\nQuestion: {b} True or False?\nAnswer:" acc = sum(int(max(range(2), key=lambda i: loglik(prompt(ex), " " + LABELS[i])) == ex["label"]) for ex in ds) / len(ds) print("accuracy", acc)
Pitfalls
- Labels are strings in the tsv_format config; map them to ints yourself.
- The 'neutral' class here covers both neutral and contradiction — do not merge with 3-way datasets without re-labelling.
References
- Paper (AAAI 2018)https://ojs.aaai.org/index.php/AAAI/article/view/12022
- AI2 leaderboardhttps://leaderboard.allenai.org/scitail/
- Hugging Facehttps://huggingface.co/datasets/allenai/scitail