STS-B
Tests whether a model can rate how similar two sentences are in meaning, on a scale from 0 (unrelated) to 5 (identical).
At a glance
What it tests
STS-B asks for a number instead of a label. Humans rated how similar two sentences are on a 0 (unrelated) to 5 (equivalent) scale, and a model is scored by how well its predictions correlate with those ratings.
Because it is graded, STS-B is the standard check for sentence-embedding models: cosine similarity of two embeddings should track the human score. Pearson rewards getting the magnitude right; Spearman only the ranking.
GPT-1 reached 82.0 Pearson by adding a linear regression head on the pooled representation.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned with a regression head | 82.0 Pearson | Prior best 81.0 |
State of the art
Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
sentence1, sentence2, label (float 0–5)
from datasets import load_dataset ds = load_dataset("nyu-mll/glue", "stsb", split="validation") print(ds) # columns: sentence1, sentence2, label (float 0–5) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
import re from scipy.stats import pearsonr, spearmanr preds, golds = [], [] for ex in ds: p = f"On a scale of 0 to 5, how similar are these sentences?\n1: {ex['sentence1']}\n2: {ex['sentence2']}\nScore:" m = re.search(r"\d+(\.\d+)?", generate(p, max_new_tokens=4)); preds.append(float(m.group()) if m else 2.5); golds.append(ex["label"]) print("pearson", pearsonr(preds, golds)[0], "spearman", spearmanr(preds, golds)[0])
Pitfalls
- Report both Pearson and Spearman; embedding papers usually quote Spearman×100.
- For a generative LLM, prompt for a number and parse it — or score with cosine similarity of hidden states; the two give different results.
- The Hub's `sentence-transformers/stsb` copy normalises scores to 0–1.
References
- Paper (arXiv 1708.00055)https://arxiv.org/abs/1708.00055
- GLUEhttps://gluebenchmark.com/
- Hugging Face (GLUE/stsb)https://huggingface.co/datasets/nyu-mll/glue