WiC
Tests whether a model can tell if the same word means the same thing in two different sentences — 'bed' of a river versus 'bed' you sleep in.
At a glance
What it tests
'There's a lot of trash on the bed of the river' versus 'I keep a glass of water next to my bed' — same word, different meaning. WiC gives two sentences containing the same target word and asks whether the sense is the same.
It is a test of whether a model's representation of a word changes with context, which is exactly what contextual embeddings promised. Humans reach 80%.
It has turned out to be surprisingly resistant: GPT-3 scored 49.4% few-shot (chance is 50%), and even fine-tuned ensembles top out around 78%, making WiC the one SuperGLUE task still below the human baseline.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Few-shot (32 examples), SuperGLUE | 49.4 acc | Zero-shot 0.0 reported (format failure); fine-tuned SOTA then 76.1 |
State of the art
Vega v2 and PaLM 540B 77.4. Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
word, sentence1, sentence2, start1/end1/start2/end2 (character offsets of the word), label (0 different · 1 same)
from datasets import load_dataset ds = load_dataset("aps/super_glue", "wic", split="validation") print(ds) # columns: word, sentence1, sentence2, start1/end1/start2/end2 (character offsets of the word), label print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
# Binary pair classification: compare log P(" yes") vs log P(" no") def prompt(ex): k = [c for c in ex if c not in ("label", "idx", "word")][:2] return f"Sentence 1: {ex[k[0]]}\nSentence 2: {ex[k[1]]}\nQuestion: Do these mean the same thing? yes or no?\nAnswer:" acc = sum(int((loglik(prompt(ex), " yes") > loglik(prompt(ex), " no")) == bool(ex["label"])) for ex in ds) / len(ds) print("accuracy", acc)
Task name(s): wic. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks wic --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Always compare to 50% — many few-shot setups are at chance.
- Prompt wording matters enormously here; GPT-3's failure was partly a prompt-format issue.
- Target-word offsets let you mark the word in the prompt (e.g. with quotes), which usually helps.
References
- Paper (arXiv 1808.09121)https://arxiv.org/abs/1808.09121
- Dataset sitehttps://pilehvar.github.io/wic/
- Hugging Face (super_glue/wic)https://huggingface.co/datasets/aps/super_glue