Generation

CNN / Daily Mail

CNN / Daily Mail summarization

Tests whether a model can summarise a news article into a few bullet points, scored by overlap with the editor's own summary.

Released 2015 Metric ROUGE-1 / 2 / L SOTA 47.78 / 23.55 / 44.57 · BRIO (2022)
01

At a glance

Task
Abstractive summarisation (article → 3–4 highlight sentences)
Size
287k train / 13k valid / 11.5k test
Metric
ROUGE-1 / ROUGE-2 / ROUGE-L F1
Source
CNN and Daily Mail articles with editor-written bullet-point highlights
Versions
1.0.0 (anonymised entities), 2.0.0, 3.0.0 (raw text — use this)
Teaching Machines to Read and Comprehend (QA version); Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond (summarization version)
Hermann et al. 2015; Nallapati et al. 2016 · NeurIPS 2015 / CoNLL 2016
Read the paper ↗ Hugging Face ↗
02

What it tests

Every CNN and Daily Mail article used to come with a few bullet points summarising it. Pair the article with its bullets and you have a summarisation dataset of 300k examples — large enough to train sequence-to-sequence models, which is why it became the default.

ROUGE counts n-gram overlap between the model's summary and the editor's bullets. It rewards summaries that reuse the article's wording, and the reference bullets are quite extractive (lead-3, copying the first three sentences, scores ~40 ROUGE-1).

GPT-2 tried it zero-shot by appending 'TL;DR:' to the article and taking the first three generated sentences. It scored 26.6 ROUGE-L — below lead-3 — but produced readable summaries, and the 'TL;DR:' trick was an early example of prompting.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Article (excerpt)
(CNN) — The Palestinian Authority officially became the 123rd member of the International Criminal Court on Wednesday, a step that gives the court jurisdiction over alleged crimes in Palestinian territories. The formal accession was marked with a ceremony at The Hague…
Highlights (reference)
Membership gives the ICC jurisdiction over alleged crimes committed in Palestinian territories since last June. · Israel and the United States opposed the move, which could open the door to war crimes investigations against Israelis.
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot, 'TL;DR:' prompt, top-k=2 sampling, first 3 sentencesR-1 29.34R-2 8.27R-L 26.58Lead-3 40.38 / 17.44 / 36.62; SOTA then 40.98 / 18.28 / 37.33 (Bottom-Up). Removing 'TL;DR:' cost 6.4 ROUGE points
05

State of the art

Best known · ROUGE-1 / 2 / L
47.78 / 23.55 / 44.57
no human baseline
Model
BRIO (Liu et al., ACL 2022)
2022
Human baseline
n/a

ROUGE SOTA has stalled; LLM summaries score lower on ROUGE but win human preference studies (Goyal et al. 2022). Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

article, highlights, id

python · load
from datasets import load_dataset

ds = load_dataset("abisee/cnn_dailymail", "3.0.0", split="test")
print(ds)            # columns: article, highlights, id
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
import evaluate
rouge = evaluate.load("rouge")
preds, refs = [], []
for ex in ds.select(range(500)):
    out = generate(ex["article"][:6000] + "\nTL;DR:", max_new_tokens=100)
    preds.append(" ".join(out.split(". ")[:3])); refs.append(ex["highlights"])
print(rouge.compute(predictions=preds, references=refs, use_stemmer=True))
lead3 = [" ".join(ex["article"].split(". ")[:3]) for ex in ds.select(range(500))]
print("lead-3 baseline", rouge.compute(predictions=lead3, references=refs, use_stemmer=True))
07

Pitfalls

  • Use version 3.0.0 (raw text); 1.0.0 has anonymised @entity tokens.
  • Report ROUGE with the standard `rouge-score` package, stemming on, and say whether you used F1 (standard) or recall.
  • Always include lead-3 as a baseline — it beats most zero-shot LLMs on ROUGE.
08

References

All benchmarks