RTE
Tests whether a model can decide if a short news passage implies a hand-written statement; a small, old and surprisingly hard set.
At a glance
What it tests
RTE predates deep learning. The premise is a sentence or short passage from a news article; the hypothesis is a sentence written by a human; the question is whether a typical reader would infer the hypothesis from the premise.
Two things make it hard despite its age: the training set is tiny (about 2,500 pairs), and the hypotheses were written by experts rather than crowd-workers, so there are fewer cheap lexical cues. GPT-1 only reached 56% on it, barely above the 50% majority baseline.
With large pre-trained models the picture flipped — RTE became one of the tasks where scale helps most, which is why GPT-3 reported it as part of SuperGLUE.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned | 56.0 acc | Below prior best 61.7; the only GLUE task GPT-1 lost |
| Few-shot (32 examples in context), SuperGLUE version | 69.0 acc | Zero-shot 63.5, one-shot 70.4; fine-tuned SOTA at the time 92.5 |
State of the art
GLUE-leaderboard RTE best is 94.1 (Turing NLR v5). Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
premise, hypothesis, label (0 entailment · 1 not_entailment). The GLUE copy is `nyu-mll/glue`, config `rte`.
from datasets import load_dataset ds = load_dataset("aps/super_glue", "rte", split="validation") print(ds) # columns: premise, hypothesis, label (0 entailment · 1 not_entailment) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
LABELS = ["True", "False"] # entailment / not_entailment def prompt(ex): a = ex.get("premise") or ex.get("question"); b = ex.get("hypothesis") or ex.get("sentence") return f"{a}\nQuestion: {b} True or False?\nAnswer:" acc = sum(int(max(range(2), key=lambda i: loglik(prompt(ex), " " + LABELS[i])) == ex["label"]) for ex in ds) / len(ds) print("accuracy", acc)
Task name(s): rte. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks rte --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- 277 dev examples means ±3 points of noise between runs; average several seeds.
- GLUE and SuperGLUE ship the same data with different split sizes; cite which one.
References
- RTE1 overviewhttps://link.springer.com/chapter/10.1007/11736790_9
- SuperGLUE paperhttps://arxiv.org/abs/1905.00537
- Hugging Face (super_glue/rte)https://huggingface.co/datasets/aps/super_glue