NLI

CommitmentBank (CB)

CommitmentBank

Tests whether a model can tell how strongly a speaker believes a clause embedded in their sentence — for example, after 'I doubt that…'.

Released 2019 Metric SuperGLUE test SOTA 98.6 F1 / 99.2 acc · Vega v2 (2022)
01

At a glance

Task
3-way (entailment / contradiction / neutral)
Size
250 train / 56 dev / 250 test
Metric
Accuracy and macro-F1
Source
WSJ, British National Corpus fiction, Switchboard dialogue
Part of
SuperGLUE
The CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, Simons, Tonhauser · Sinn und Bedeutung 23, 2019
Read the paper ↗ Hugging Face ↗
02

What it tests

CB looks at a specific linguistic phenomenon: when a sentence embeds a clause ('I don't think he will come'), how strongly does the speaker commit to the embedded clause ('he will come')? The premise is a short discourse ending in such a sentence; the hypothesis is the embedded clause.

Because the premises are naturally occurring text and the judgement depends on negation, modality and the embedding verb, surface overlap is useless — the hypothesis is literally contained in the premise every time.

With only 250 training examples, it is also a test of how well a model can use its prior knowledge rather than fit the task.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Premise
It was a complex language. Not written down but handed down. One might say it was peeled down.
Hypothesis
the language was peeled down
Answerentailment
Premise
B: And yet, uh, I we-, I hope to see employer based, you know, helping out. You know, child, uh, care centers at the place of employment and things like that, that will help out. A: Uh-huh. B: What do you think, do you think we are, setting a trend?
Hypothesis
they are setting a trend
Answerneutral
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Few-shot (32 examples), SuperGLUE75.6 acc / 52.0 F1Zero-shot 46.4 acc; fine-tuned SOTA 96.9 acc
05

State of the art

Best known · SuperGLUE test
98.6 F1 / 99.2 acc
✓ saturated · at or above human level
Model
Vega v2
2022
Human baseline
95.8 F1 / 98.9 acc

Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

premise, hypothesis, label (0 entailment · 1 contradiction · 2 neutral — note the order differs from SNLI/MNLI)

python · load
from datasets import load_dataset

ds = load_dataset("aps/super_glue", "cb", split="validation")
print(ds)            # columns: premise, hypothesis, label (0 entailment · 1 contradiction · 2 neutral — note the order di
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
# Score each label by the log-likelihood of its verbaliser after the pair (GPT-3 style)
LABELS = ["True", "Neither", "False"]      # entailment / neutral / contradiction
def prompt(ex): return f"{ex['premise']}\nQuestion: {ex['hypothesis']} True, False, or Neither?\nAnswer:"
correct = 0
for ex in ds:
    if ex["label"] < 0: continue
    scores = [loglik(prompt(ex), " " + lab) for lab in LABELS]
    correct += int(scores.index(max(scores)) == ex["label"])
print("accuracy", correct / len(ds))
4Or use lm-evaluation-harness

Task name(s): cb. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks cb --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Label order is entailment/contradiction/neutral, not entailment/neutral/contradiction. Never reuse an MNLI label map.
  • 56 dev examples: a single example is ~1.8 points. Report F1 alongside accuracy because the classes are very imbalanced (few contradictions).
08

References

All benchmarks