WebQuestions
Tests whether a model can answer short factual questions mined from Google search suggestions, from memory alone.
At a glance
What it tests
WebQuestions was built for semantic parsing — turning a question into a database query — but the GPT-3 authors reused it as a closed-book factoid test: can the model answer 'what language do people in the czech republic speak?' from memory?
The questions are short, entity-centric and often have several valid answers (all the languages, all the children), which makes exact match harsh.
GPT-3 jumped from 14.4% zero-shot to 41.5% few-shot — one of the largest zero-to-few-shot gains in the paper, because the examples teach the model the expected answer format more than the facts.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Few-shot (64 examples), closed-book | 41.5 EM | Zero-shot 14.4, one-shot 25.3; SOTA then 45.5 (RAG, open-domain), 37.4 (T5-11B closed-book) |
State of the art
GenRead 54.4, COMBO 54.2. Few-shot PaLM 540B 43.5 (64-shot). Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
url (Freebase entity), question, answers (list)
from datasets import load_dataset ds = load_dataset("stanfordnlp/web_questions", split="test") print(ds) # columns: url (Freebase entity), question, answers (list) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
import re, string def norm(s): s = s.lower(); s = "".join(c for c in s if c not in string.punctuation) s = re.sub(r"\b(a|an|the)\b", " ", s); return " ".join(s.split()) FEW = "Q: who wrote the novel 1984?\nA: George Orwell\n\nQ: what is the capital of australia?\nA: Canberra\n\n" em = 0 for ex in ds: golds = ex.get("answer") or ex.get("answers") or ex["answer"]["aliases"] pred = generate(FEW + f"Q: {ex['question']}\nA:", max_new_tokens=16, stop="\n") em += int(norm(pred) in {norm(g) for g in golds}) print("exact match", em / len(ds))
Task name(s): webqs. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks webqs --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Multiple gold answers are common; a prediction matching any one of them counts in the GPT-3 convention.
- Only 3.8k training examples; few-shot results vary with which examples you sample.
- Answers reflect Freebase circa 2013.
References
- Paper (EMNLP 2013)https://aclanthology.org/D13-1160/
- Hugging Facehttps://huggingface.co/datasets/stanfordnlp/web_questions