Benchmark Suites

SuperGLUE

SuperGLUE

A harder collection of eight language-understanding tests (yes/no questions, cause and effect, word sense, coreference and more) with one overall score.

Released May 2019 Metric overall leaderboard score SOTA 91.3 · Vega v2 (2022)
01

At a glance

Tasks
BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC (+ diagnostic sets AX-b, AX-g)
Score
Average of per-task scores (F1/acc averaged for CB, MultiRC, ReCoRD)
Test labels
Hidden; submission via super.gluebenchmark.com
Human baseline
89.8
Status
Passed by models in 2020–21; leaderboard static since 2022
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, Pruksachatkun, Nangia, Singh, Michael, Hill, Levy, Bowman · NeurIPS 2019
Read the paper ↗ Hugging Face ↗
02

What it tests

When GLUE saturated, the same group chose eight tasks that BERT still did badly on, with more varied formats — yes/no QA, causal choice, multi-answer comprehension, cloze, word sense, coreference — and a harder human baseline.

GPT-3 reported SuperGLUE as its headline comparison with fine-tuned models: 71.8 few-shot (32 examples per task, all in context) against 89.0 for a fine-tuned BERT-large and 89.3 for a fine-tuned SOTA. The per-task breakdown shows where in-context learning was strong (COPA, ReCoRD) and where it collapsed (WiC, CB, MultiRC EM).

T5 and later DeBERTa passed the human baseline in 2020–21; the top score has been 91.3 since 2022.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Task formats
BoolQ
passage + yes/no question
CB
3-way entailment on embedded clauses
COPA
cause/effect choice
MultiRC
multi-answer comprehension
ReCoRD
entity cloze over news
RTE
entailment
WiC
word-sense match
WSC
pronoun coreference
AnswerScore = mean over 8 tasks
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (32 examples/task), 175B71.8 averageFine-tuned BERT-large 69.0; fine-tuned BERT++ 71.5; fine-tuned SOTA 89.3; human 89.8
05

State of the art

Best known · overall leaderboard score
91.3
✓ saturated · at or above human level
Model
Vega v2 (JD Explore)
2022
Human baseline
89.8

ST-MoE-32B 91.2, Turing NLR v5 90.9, PaLM 540B 90.4. Source: super.gluebenchmark.com ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

Configs: boolq, cb, copa, multirc, record, rte, wic, wsc, wsc.fixed, axb, axg. `evaluate.load('super_glue', '<task>')` for metrics.

python · load
from datasets import load_dataset

ds = load_dataset("aps/super_glue", "<task>", split="validation")
print(ds)            # columns: Configs: boolq, cb, copa, multirc, record, rte, wic, wsc, wsc
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
DATASET = "aps/super_glue"
# Loop every task of the suite and compute the official metric with `evaluate`
import evaluate
TASKS = ["cola", "sst2", "mrpc", "qqp", "mnli", "qnli", "rte"]   # GLUE; for SuperGLUE use boolq, cb, copa, multirc, record, rte, wic, wsc.fixed
for t in TASKS:
    d = load_dataset(DATASET, t, split="validation" if t != "mnli" else "validation_matched")
    metric = evaluate.load(DATASET.split("/")[-1], t)
    preds = [predict(t, ex) for ex in d]      # your per-task prompting / scoring goes here
    print(t, metric.compute(predictions=preds, references=d["label"]))
4Or use lm-evaluation-harness

Task name(s): super-glue-lm-eval-v1 (group: boolq, cb, copa, multirc, record, rte, wic, wsc). The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks super-glue-lm-eval-v1 --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Use wsc.fixed, not wsc.
  • lm-eval's SuperGLUE group scores by log-likelihood; T5-style prompted generation gives different per-task numbers.
  • Averaging convention: CB uses mean(acc, F1), MultiRC mean(F1a, EM), ReCoRD mean(F1, EM).
08

References

All benchmarks