Benchmark Suites

GLUE

General Language Understanding Evaluation

A collection of nine sentence-understanding tests (grammar, sentiment, paraphrase, similarity and inference) combined into one overall score.

Released Apr 2018 Metric overall leaderboard score SOTA 91.3 · Vega v1 (2022)
01

At a glance

Tasks
CoLA, SST-2, MRPC, STS-B, QQP, MNLI (m/mm), QNLI, RTE, WNLI, plus a diagnostic NLI set
Score
Unweighted average of per-task metrics (MCC for CoLA, F1/acc for MRPC & QQP, Pearson/Spearman for STS-B, accuracy elsewhere)
Test labels
Hidden; evaluated by submitting predictions to gluebenchmark.com
Human baseline
87.1
Status
Leaderboard frozen since ~2022; effectively retired
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, Singh, Michael, Hill, Levy, Bowman · ICLR 2019
Read the paper ↗ Hugging Face ↗
02

What it tests

GLUE bundled nine existing datasets into one number so that a single model could be compared across tasks. All of them are sentence or sentence-pair classification, which is why the GPT-1 recipe — one transformer, a tiny task-specific head, inputs concatenated with delimiters — fit so well.

GPT-1 scored 72.8 on the leaderboard in June 2018, the previous best being 68.9. BERT reached 80.5 four months later, and by 2019 models passed the 87.1 human baseline; SuperGLUE was created because GLUE had stopped discriminating between models.

WNLI (a Winograd-derived task with a broken train/dev split) is usually predicted as majority class; most papers footnote this.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

One task per row
Single-sentence
CoLA (acceptability), SST-2 (sentiment)
Similarity / paraphrase
MRPC, QQP, STS-B
Inference
MNLI, QNLI, RTE, WNLI
AnswerScore = mean over tasks
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Fine-tuned per task, single model72.8 overallPrior best 68.9; wins on 8 of 9 tasks (not RTE/SST-2 vs ensembles)
05

State of the art

Best known · overall leaderboard score
91.3
✓ saturated · at or above human level
Model
Vega v1 (JD Explore) · Turing ULR v6 (Microsoft)
2022
Human baseline
87.1

Leaderboard has not moved since 2022. Source: gluebenchmark.com ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

One config per task: cola, sst2, mrpc, qqp, stsb, mnli, qnli, rte, wnli, ax. `evaluate.load('glue', '<task>')` gives the matching metric.

python · load
from datasets import load_dataset

ds = load_dataset("nyu-mll/glue", "<task>", split="validation")
print(ds)            # columns: One config per task: cola, sst2, mrpc, qqp, stsb, mnli, qnli, rte, wnli, ax
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
DATASET = "nyu-mll/glue"
# Loop every task of the suite and compute the official metric with `evaluate`
import evaluate
TASKS = ["cola", "sst2", "mrpc", "qqp", "mnli", "qnli", "rte"]   # GLUE; for SuperGLUE use boolq, cb, copa, multirc, record, rte, wic, wsc.fixed
for t in TASKS:
    d = load_dataset(DATASET, t, split="validation" if t != "mnli" else "validation_matched")
    metric = evaluate.load(DATASET.split("/")[-1], t)
    preds = [predict(t, ex) for ex in d]      # your per-task prompting / scoring goes here
    print(t, metric.compute(predictions=preds, references=d["label"]))
4Or use lm-evaluation-harness

Task name(s): glue (group: cola, sst2, mrpc, qqp, mnli, qnli, rte, wnli). The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks glue --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Test labels are hidden; dev-set 'GLUE scores' are common in papers but not comparable to leaderboard numbers.
  • Averaging requires the exact per-task metric mix; use `evaluate`'s glue metric per task.
  • WNLI is adversarial by accident — majority-class (65.1) beats most models.
08

References

All benchmarks