QQP
Tests whether a model can tell if two questions asked on Quora are asking the same thing.
At a glance
What it tests
Quora merges questions that ask the same thing. QQP is their labelled history of which pairs were merged: 'How can I be a good geologist?' and 'What should I do to be a great geologist?' are duplicates; questions that merely share a topic are not.
The task is paraphrase detection at scale with noisy labels (Quora's own process was imperfect), and it is also naturally imbalanced, so F1 is the headline number.
GPT-1 reported 70.3 F1 — a large jump from the 66.1 of the prior GLUE baseline — and fine-tuned models now exceed the human F1 of 59.5 by almost twenty points, mostly because the humans in the baseline did not see the labelling guidelines.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned | 70.3 F1 | Prior best 66.1 |
State of the art
GLUE test F1 is on a harder, differently balanced test set than the public Kaggle split. Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
question1, question2, label (0 not_duplicate · 1 duplicate)
from datasets import load_dataset ds = load_dataset("nyu-mll/glue", "qqp", split="validation") print(ds) # columns: question1, question2, label (0 not_duplicate · 1 duplicate) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
# Binary pair classification: compare log P(" yes") vs log P(" no") def prompt(ex): k = [c for c in ex if c not in ("label", "idx", "word")][:2] return f"Sentence 1: {ex[k[0]]}\nSentence 2: {ex[k[1]]}\nQuestion: Do these mean the same thing? yes or no?\nAnswer:" acc = sum(int((loglik(prompt(ex), " yes") > loglik(prompt(ex), " no")) == bool(ex["label"])) for ex in ds) / len(ds) print("accuracy", acc)
Task name(s): qqp. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks qqp --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Labels are noisy; a few points of F1 are irreducible.
- The GLUE test set is not the Kaggle test set; numbers from Kaggle competitions are not comparable.
- Validation is 40k rows — subsample for quick few-shot checks.
References
- Quora releasehttps://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs
- GLUEhttps://gluebenchmark.com/
- Hugging Face (GLUE/qqp)https://huggingface.co/datasets/nyu-mll/glue