Generation

WMT Translation

WMT14 En↔Fr, WMT16 En↔De, WMT16 En↔Ro

Tests how well a model translates news sentences between English and French, German or Romanian, scored by overlap with human translations.

Released 2014–2016 Metric BLEU SOTA Fr→En 39.2 (GPT-3) · En→Fr 45.6 (Edunov 2018) · De→En 40.6 (GPT-3) · En→De 41.2 (Wang 2018) · Ro→En 39.9 (mBART) · En→Ro 38.5 (mBART) · see per-direction (2020)
01

At a glance

Task
Sentence-level translation
Test sets
newstest2014 (Fr–En, 3,003 sentences), newstest2016 (De–En 2,999; Ro–En 1,999)
Metric
BLEU (GPT-3 used SacreBLEU with intl tokenisation; older papers use multi-bleu)
Source
News articles translated by professionals
Training data
Europarl, CommonCrawl, news-commentary, UN — but the GPT papers use none of it
Findings of the 2014 / 2016 Workshop on Statistical Machine Translation
Bojar et al. · WMT 2014, WMT 2016
Read the paper ↗ Hugging Face ↗
02

What it tests

WMT is the annual shared task for machine translation; its test sets are the standard yardstick. BLEU measures n-gram overlap with reference translations, and 'supervised SOTA' means systems trained on millions of parallel sentence pairs.

GPT-2's experiment was almost a stunt: its training data was English-only (with some French text that slipped in), and prompting 'english sentence = french sentence' pairs got 11.5 BLEU on Fr→En — worse than unsupervised MT, but non-zero. GPT-3 pushed this much further: 39.2 BLEU Fr→En few-shot beat the supervised SOTA, and De→En and Ro→En did the same, while translation *into* the other languages lagged because the model is mostly English.

The lesson that stuck: a big enough English model learns other languages as a side effect, and it is much better at reading them than writing them.

03

Examples

Items from the dataset (or representative items where the originals are not freely redistributable).

Few-shot prompt format (GPT-3)
Prompt
French: Je suis étudiant. = English: I am a student.\nFrench: Où est la gare ? = English:
Model output
Where is the train station?
AnswerBLEU against the reference English
newstest2014 Fr→En sample
Source
Spectaculaire saut en "wingsuit" au-dessus de Bogota
Reference
Spectacular Wingsuit Jump Over Bogota
04

Reported results

Scores reported on this benchmark, with the evaluation setting each one used.

SettingReportedContext
Zero-shot with example pairs in context; English-only training dataFr→En 11.5 BLEUEn→Fr 5 BLEUUnsupervised SOTA 33.5; the paper found only 10 MB of French in WebText
Few-shot (64 examples)Fr→En 39.2En→Fr 32.6De→En 40.6En→De 29.7Ro→En 39.5En→Ro 21.0Supervised SOTA: 35.0 / 45.6 / 40.2 / 41.2 / 39.9 / 38.5; GPT-3 beat it on all three X→En directions
05

State of the art

Best known · BLEU
Fr→En 39.2 (GPT-3) · En→Fr 45.6 (Edunov 2018) · De→En 40.6 (GPT-3) · En→De 41.2 (Wang 2018) · Ro→En 39.9 (mBART) · En→Ro 38.5 (mBART)
no human baseline
Model
see per-direction
2020
Human baseline
n/a

Figures as tabulated in GPT-3 Table 3.4. These test sets are no longer the research frontier (WMT moves on yearly; COMET has replaced BLEU), so later numbers are sparse. Source: arxiv.org ↗ · checked October 2026.

06

Quickstart

Load the benchmark and score a Hugging Face causal LM on it in a few lines.

1Install
shell
pip install -U datasets transformers evaluate lm-eval sacrebleu rouge-score scikit-learn
2Load the data

translation (dict with 'en' and 'fr' / 'de' / 'ro'). WMT16 configs: `wmt/wmt16`, 'de-en' and 'ro-en'.

python · load
from datasets import load_dataset

ds = load_dataset("wmt/wmt14", "fr-en", split="test")
print(ds)            # columns: translation (dict with 'en' and 'fr' / 'de' / 'ro')
print(ds[0])
3Score a model

Two helpers cover every benchmark in this collection: loglik for anything multiple-choice (score each option, pick the highest) and generate for anything free-form. Swap MODEL for whatever you are evaluating.

python · helpers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL = "gpt2"                                   # or "Qwen/Qwen2.5-1.5B", "meta-llama/Llama-3.2-1B", …
tok = AutoTokenizer.from_pretrained(MODEL); model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).eval().to("cuda")

@torch.no_grad()
def loglik(context, continuation):
    """Sum of log-probs of `continuation` tokens given `context` — the standard multiple-choice scorer."""
    ids = tok(context + continuation, return_tensors="pt").input_ids.to(model.device)
    n_ctx = len(tok(context).input_ids)
    logp = model(ids).logits.log_softmax(-1)[0, :-1]
    return logp.gather(1, ids[0, 1:, None])[n_ctx - 1:].sum().item()

@torch.no_grad()
def generate(prompt, max_new_tokens=32, stop=None):
    ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False, pad_token_id=tok.eos_token_id)
    text = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    return text.split(stop)[0] if stop else text
python · evaluate
import sacrebleu
src, tgt = "fr", "en"                          # direction: fr -> en
FEW = "".join(f"French: {ex['translation']['fr']} = English: {ex['translation']['en']}\n" for ex in ds.select(range(5)))
hyps, refs = [], []
for ex in ds.select(range(5, 505)):
    out = generate(FEW + f"French: {ex['translation'][src]} = English:", max_new_tokens=96, stop="\n")
    hyps.append(out.strip()); refs.append(ex["translation"][tgt])
print(sacrebleu.corpus_bleu(hyps, [refs]))     # report the signature too
4Or use lm-evaluation-harness

Task name(s): wmt14-en-fr, wmt14-fr-en, wmt16-en-de, wmt16-de-en, wmt16-en-ro, wmt16-ro-en. The harness reproduces the GPT-3 prompt formats and reports the standard metric.

shell
lm_eval --model hf --model_args pretrained=gpt2,dtype=bfloat16 \
        --tasks wmt14-en-fr --num_fewshot 0 --batch_size auto --output_path results/
07

Pitfalls

  • Use SacreBLEU and report its signature; multi-bleu numbers are 1–2 points different.
  • Translation *into* English is much easier for English-centric LLMs; report both directions.
  • The test sets are tiny news sets from 2014/2016 — contamination is likely for web-trained models.
08

References

All benchmarks