mlsanityData to 14 Oct 2026

BLiMP: The Benchmark of Linguistic Minimal Pairs for English

arXiv 1912.00582 ↗First posted 2 Dec 2019cs.CLcited by 564

Not on the map. Why?

Abstract (arXiv)

We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars, and aggregate human agreement with the labels is 96.4%. We use it to evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs. We find that state-of-the-art models identify morphological contrasts reliably, but they struggle with semantic restrictions on the distribution of quantifiers and negative polarity items and subtle syntactic phenomena such as extraction islands.

Key facts. Cited by 564 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 44; 2022: 44; 2023: 80; 2024: 88; 2025: 152; 2026: 156.

564 papers.

Sort:

By area:

  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  3. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  4. 2021MetaICL: Learning to Learn In Contextcited by 454
  5. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  6. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  7. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  8. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  9. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  10. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  11. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  12. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
38 more
  1. 2025Agentic Large Language Models, a surveycited by 108
  2. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 99
  3. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  4. 2022Interpreting Language Models with Contrastive Explanationscited by 92
  5. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  6. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  7. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  8. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  9. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  10. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 59
  11. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 59
  12. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  13. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  14. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  15. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  16. 2023Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformationcited by 51
  17. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  18. 2024Language models align with human judgments on key grammatical constructionscited by 49
  19. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  20. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  21. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  22. 2021The Zero Resource Speech Challenge 2021: Spoken language modellingcited by 47
  23. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  24. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  25. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 45
  26. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  27. 2023Prompting is not a substitute for probability measurements in large language modelscited by 44
  28. 2023Quantifying Context Mixing in Transformerscited by 43
  29. 2023Explaining How Transformers Use Context to Build Predictionscited by 43
  30. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  31. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 42
  32. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  33. 2025People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated textcited by 40
  34. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  35. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
  36. 2021The Grammar-Learning Trajectories of Neural Language Modelscited by 36
  37. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  38. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35

50 most cited of 564 All 564 (JSON)