SST-2
Tests whether a model can tell if a movie-review sentence is positive or negative.
At a glance
What it tests
SST started as a treebank: every phrase in every parse tree of 11,855 movie-review sentences has a sentiment label, so a model can be trained on how sentiment composes ('not bad' from 'bad'). SST-2 keeps only full sentences, drops the neutral ones and collapses the five-point scale to positive/negative.
It is the simplest task in GLUE — single sentences, two labels, lots of training data — and was the first to be effectively solved. It still appears in model cards as a sanity check.
GPT-1 reached 91.3%; the human baseline (97.8%) was matched by 2021.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned | 91.3 acc | Prior best 93.2 (SOTA then was an ensemble); GPT-1 did not win this one |
State of the art
Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
sentence, label (0 negative · 1 positive)
from datasets import load_dataset ds = load_dataset("nyu-mll/glue", "sst2", split="validation") print(ds) # columns: sentence, label (0 negative · 1 positive) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
LABELS = {"sst2": ["negative", "positive"], "cola": ["unacceptable", "acceptable"]}["sst2"] # pick the task
def prompt(ex): return f"{ex['sentence']}\nQuestion: Is this sentence {LABELS[0]} or {LABELS[1]}?\nAnswer:"
preds = [max(range(2), key=lambda i: loglik(prompt(ex), " " + LABELS[i])) for ex in ds]
from sklearn.metrics import accuracy_score, matthews_corrcoef
print("acc", accuracy_score(ds["label"], preds), "mcc", matthews_corrcoef(ds["label"], preds))Task name(s): sst2. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks sst2 --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- The train split includes sub-phrases; dev/test are full sentences only.
- Text is lower-cased and tokenised with spaces around punctuation — keep it as is for comparability.
- Zero-shot LLM prompts should include 'positive'/'negative' as the verbaliser; 'good'/'bad' changes results.
References
- Paper (EMNLP 2013)https://aclanthology.org/D13-1170/
- Treebank sitehttps://nlp.stanford.edu/sentiment/
- Hugging Face (GLUE/sst2)https://huggingface.co/datasets/nyu-mll/glue