ANLI
Tests whether a model can judge if one sentence follows from another, using examples deliberately written to trick strong models.
CommitmentBank (CB)
Tests whether a model can tell how strongly a speaker believes a clause embedded in their sentence — for example, after 'I doubt that…'.
MultiNLI
Tests whether a model can judge if one sentence follows from another across many kinds of writing — fiction, phone calls, government reports and more.
QNLI
Tests whether a model can tell if a given sentence from Wikipedia actually contains the answer to a question.
RTE
Tests whether a model can decide if a short news passage implies a hand-written statement; a small, old and surprisingly hard set.
SciTail
Tests whether a model can decide if a sentence found on the web supports a statement taken from a school science exam.
SNLI
Tests whether a model can tell if one sentence logically follows from, contradicts, or is unrelated to another, using short sentences about everyday photos.
BoolQ
Tests whether a model can answer real yes/no questions people typed into Google, given a Wikipedia passage.
CoQA
Tests whether a model can answer a chain of questions about a passage, where each question depends on the earlier ones.
DROP
Tests whether a model can read a paragraph and do simple arithmetic, counting or date comparison to answer a question.
MultiRC
Tests whether a model can answer questions that need information from several sentences, where more than one option can be correct.
QuAC
Tests whether a model can answer a student's questions about a Wikipedia article turn by turn, including saying when there is no answer.
RACE
Tests reading comprehension using multiple-choice questions from English exams given to Chinese school students.
ReCoRD
Tests whether a model can fill a missing name or place in a sentence about a news article, which needs some common sense.
SQuAD 2.0
Tests whether a model can find the answer to a question in a Wikipedia paragraph, or recognise when the paragraph does not contain it.
Natural Questions
Tests whether a model can answer real Google search questions from memory, without being shown any document.
TriviaQA
Tests how many trivia-quiz questions a model can answer from the facts it memorised during training.
WebQuestions
Tests whether a model can answer short factual questions mined from Google search suggestions, from memory alone.
ARC
Tests whether a model can answer grade-school science exam questions; the Challenge split contains the ones simple word-matching cannot solve.
COPA
Tests causal common sense: given a short statement, which of two alternatives is the more likely cause or effect.
HellaSwag
Tests whether a model can pick the most sensible continuation of an everyday scene from four options written to fool it.
OpenBookQA
Tests whether a model can combine a given science fact with everyday knowledge to answer a question.
PIQA
Tests physical common sense: given an everyday goal, which of two near-identical instructions actually works in the real world.
Story Cloze
Tests whether a model can choose the right ending for a short five-sentence story about everyday life.
Winograd Schema Challenge
Tests whether a model can work out who a pronoun refers to in sentences where only common sense decides — 'it' in 'the trophy does not fit in the suitcase because it is too big'.
WinoGrande
Tests whether a model can fill a blank in a sentence with the right of two people or objects, using common sense about the situation.
Children's Book Test
Tests whether a model can fill a missing word in a sentence from a children's book after reading the twenty sentences before it.
enwik8
Measures how well a model predicts raw Wikipedia text one character at a time, scored as bits per character; lower is better.
LAMBADA
Tests whether a model can guess the last word of a passage from a novel when that word can only be predicted from the whole passage.
One Billion Word
Measures how well a model predicts the next word in a billion words of shuffled news sentences, with no context between sentences.
Penn Treebank
Measures how well a model predicts the next word in Wall Street Journal text that has been simplified to a 10,000-word vocabulary.
text8
Measures how well a model predicts cleaned-up Wikipedia text (lower-case letters and spaces only) one character at a time.
WikiText-103
Measures how well a model predicts the next word in Wikipedia articles (large version with long articles); lower perplexity is better.
WikiText-2
Measures how well a model predicts the next word in Wikipedia articles (small version); lower perplexity means better prediction.
CoLA
Tests whether a model can tell if a sentence is grammatically acceptable English, using examples from linguistics textbooks.
MRPC
Tests whether a model can tell if two news sentences say the same thing in different words.
QQP
Tests whether a model can tell if two questions asked on Quora are asking the same thing.
SST-2
Tests whether a model can tell if a movie-review sentence is positive or negative.
STS-B
Tests whether a model can rate how similar two sentences are in meaning, on a scale from 0 (unrelated) to 5 (identical).
WiC
Tests whether a model can tell if the same word means the same thing in two different sentences — 'bed' of a river versus 'bed' you sleep in.
GLUE
A collection of nine sentence-understanding tests (grammar, sentiment, paraphrase, similarity and inference) combined into one overall score.
SuperGLUE
A harder collection of eight language-understanding tests (yes/no questions, cause and effect, word sense, coreference and more) with one overall score.
CNN / Daily Mail
Tests whether a model can summarise a news article into a few bullet points, scored by overlap with the editor's own summary.
WMT Translation
Tests how well a model translates news sentences between English and French, German or Romanian, scored by overlap with human translations.
GPT-3 synthetic tasks
A set of made-up probe tasks — multi-digit arithmetic, unscrambling letters, SAT-style word analogies and spotting machine-written news — used to test learning from examples in the prompt.