CoLA
Tests whether a model can tell if a sentence is grammatically acceptable English, using examples from linguistics textbooks.
At a glance
What it tests
Linguistics papers are full of example sentences, some marked with an asterisk meaning 'a native speaker would not say this'. CoLA collects those sentences and their marks: 'The book was written by John' is fine; 'Book the was written by John' is not; 'The more you would want, the less you would eat' is the kind of subtle case that makes the task interesting.
Because 70% of sentences are acceptable, accuracy is misleading; Matthews correlation (which is 0 for any constant predictor) is used instead.
GPT-1 scored 45.4 MCC, a ten-point jump over the previous best and the paper's largest relative gain on GLUE — evidence that a language model picks up grammaticality as a side effect of predicting text.
Examples
Items from the dataset (or representative items where the originals are not freely redistributable).
Reported results
Scores reported on this benchmark, with the evaluation setting each one used.
| Setting | Reported | Context |
|---|---|---|
| Fine-tuned | 45.4 MCC | Prior best 35.0 |
State of the art
Source: arxiv.org ↗ · checked October 2026.
Quickstart
Load the benchmark and score a Hugging Face causal LM on it in a few lines.
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
sentence, label (0 unacceptable · 1 acceptable)
from datasets import load_dataset ds = load_dataset("nyu-mll/glue", "cola", split="validation") print(ds) # columns: sentence, label (0 unacceptable · 1 acceptable) print(ds[0])
Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.
import torch from transformers import AutoTokenizer, AutoModelForCausalLM MODEL = "gpt2" # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", … tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda") @torch.no_grad() def loglik(context, continuation): """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer.""" ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device) n_ctx = len(tok(context).input_ids) logp = model(ids).logits.log_softmax(-1)[0, :-1] return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item() @torch.no_grad() def generate(prompt, max_new_tokens=32, stop=None): ids = tok(prompt, return_tensors="pt").input_ids.to(model.device) out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id) text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True) return text.split(stop)[0] if stop else text
LABELS = {"sst2": ["negative", "positive"], "cola": ["unacceptable", "acceptable"]}["sst2"] # pick the task
def prompt(ex): return f"{ex['sentence']}\nQuestion: Is this sentence {LABELS[0]} or {LABELS[1]}?\nAnswer:"
preds = [max(range(2), key=lambda i: loglik(prompt(ex), " " + LABELS[i])) for ex in ds]
from sklearn.metrics import accuracy_score, matthews_corrcoef
print("acc", accuracy_score(ds["label"], preds), "mcc", matthews_corrcoef(ds["label"], preds))Task name(s): cola. The harness reproduces the GPT-3 prompt formats and reports the standard metric.
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
--tasks cola --num_fewshot 0 --batch_size auto --output_path results/Pitfalls
- Use MCC, not accuracy (`sklearn.metrics.matthews_corrcoef`).
- The dev set has in-domain and out-of-domain halves (sources seen/unseen in training); the paper reports both.
- Judgements are the linguists' own and sometimes contested — expect a noise ceiling.
References
- Paper (arXiv 1805.12471)https://arxiv.org/abs/1805.12471
- Dataset sitehttps://nyu-mll.github.io/CoLA/
- Hugging Face (GLUE/cola)https://huggingface.co/datasets/nyu-mll/glue