BoolQ
Tests whether a model can answer real yes/no questions people typed into Google, given a Wikipedia passage.
At a glance
What it tests
People type a surprising number of yes/no questions into search engines ('is france the same timezone as the uk'). BoolQ collects those and pairs each with the Wikipedia paragraph a human used to answer it.
The questions look trivial but the answers require inference: the passage rarely says 'yes' or 'no' explicitly, so the model has to compare what is asked with what is stated, handle hedges, and sometimes do a bit of world knowledge.
Majority class (yes) is about 62%, which is why early models looked better than they were. GPT-3 few-shot reached 76.4%, well below fine-tuned models.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Few-shot (32 examples), SuperGLUE | 76.4 acc | Zero-shot 60.5; fine-tuned SOTA then 91.0 |
State of the art
PaLM 540B fine-tuned 91.9. Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
question, passage, label (0 no · 1 yes). Also available as `google/boolq` with boolean `answer`.
from datasets import load_dataset ds = load_dataset("aps/super_glue", "boolq", split="validation") print(ds) # columns: question, passage, label (0 no · 1 yes) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
def prompt(ex): return f"{ex['passage']}\nQuestion: {ex['question']}?\nAnswer:" acc = sum(int((loglik(prompt(ex), " yes") > loglik(prompt(ex), " no")) == bool(ex["label"])) for ex in ds) / len(ds) print("accuracy", acc)
Task name(s): boolq. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks boolq --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Always compare to the 62% majority baseline.
- Questions are lower-case and un-punctuated search queries; do not 'clean' them, that is the task.
- lm-eval scores log P('yes') vs log P('no') after the passage; a chat-style prompt can shift results by several points.
References
- Paper (arXiv 1905.10044)https://arxiv.org/abs/1905.10044
- GitHubhttps://github.com/google-research-datasets/boolean-questions
- Hugging Face (super_glue/boolq)https://huggingface.co/datasets/aps/super_glue