mlsanityData to 14 Oct 2026

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

arXiv 2504.02768 ↗First posted 3 Apr 2025cs.CLcited by 71

Not on the map. Why?

Abstract (arXiv)

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 50 of its 83 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

50 of 83 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  4. 2018Neural Network Acceptability Judgmentscited by 958
  5. 2019How multilingual is Multilingual BERT?cited by 645
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  8. 20242 OLMo 2 Furiouscited by 340
  9. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  10. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  11. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  12. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
38 more
  1. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  2. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  3. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  4. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  5. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  6. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  7. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  8. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 110
  9. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  10. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  11. 2024Language models align with human judgments on key grammatical constructionscited by 49
  12. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  13. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  14. 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
  15. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  16. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 30
  17. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  18. 2024Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Modelscited by 24
  19. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  20. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  21. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 18
  22. 2023ScandEval: A Benchmark for Scandinavian Natural Language Processingcited by 16
  23. 2024GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languagescited by 16
  24. 2021On the Limits of Minimal Pairs in Contrastive Evaluationcited by 15
  25. 2022SLING: Sino Linguistic Evaluation of Large Language Modelscited by 15
  26. 2022Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Modelscited by 14
  27. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 14
  28. 2024A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinesecited by 13
  29. 2022UniMorph 4.0: Universal Morphologycited by 11
  30. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 11
  31. 2024Controlled Evaluation of Syntactic Knowledge in Multilingual Language Modelscited by 10
  32. 2025Linguistic Generalizations are not Rules: Impacts on Evaluation of LMscited by 10
  33. 2020Automatic Extraction of Rules Governing Morphological Agreementcited by 9
  34. 2024Multilingual Gradient Word-Order Typology from Universal Dependenciescited by 9
  35. 2024Natural Language Processing RELIES on Linguisticscited by 9
  36. 2021Evaluating the Morphosyntactic Well-formedness of Generated Textscited by 8
  37. 2020Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networkscited by 7
  38. 2023The Past, Present, and Future of Typological Databases in NLPcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 45.

71 papers.

Sort:

By area:

  1. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  2. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  3. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 20
  4. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 10
  5. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  6. 2025Minimal Pair-Based Evaluation of Code-Switchingcited by 8
  7. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 8
  8. 2025Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluationscited by 7
  9. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 7
  10. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  11. 2025Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken Englishcited by 4
  12. 2025UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactionscited by 3
38 more
  1. 2025DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errorscited by 3
  2. 2025Grow Up and Merge: Scaling Strategies for Efficient Language Adaptationcited by 3
  3. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  4. 2025UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urducited by 2
  5. 2025Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspectcited by 2
  6. 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
  7. 2025COLE: a Comprehensive Benchmark for French Language Understanding Evaluationcited by 2
  8. 2025Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Settingcited by 2
  9. 2025Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysiscited by 2
  10. 2026Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrewcited by 2
  11. 2026TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representationcited by 2
  12. 2026Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptationcited by 2
  13. 2026TokEval: A Tokenizer Evaluation Suitecited by 2
  14. 2025Multilingual Vision-Language Models, A Surveycited by 1
  15. 2025On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibilitycited by 1
  16. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  17. 2026Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategiescited by 1
  18. 2026The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoningcited by 1
  19. 2026Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretrainingcited by 1
  20. 2026Implicit Representations of Grammaticality in Language Modelscited by 1
  21. 2026Light or Full Verb? A Minimal-Pair Dataset for Probing Phraseological Competence in Language Modelscited by 1
  22. 2026ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairscited by 1
  23. 2026Linear representations of grammaticality in neural language modelscited by 1
  24. 2026MORFES: A Benchmark for Productive Inflectional Competence in Modern Greekcited by 1
  25. 2026Structural priors for data-efficient language learningcited by 1
  26. 2025Understanding Syntactic Generalization in Structure-inducing Language Modelscited by 0
  27. 2025Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretrainingcited by 0
  28. 2025Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attractioncited by 0
  29. 2025Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Researchcited by 0
  30. 2026A Study of Crosslinguistic Influence in Language Modelscited by 0
  31. 2026Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languagescited by 0
  32. 2026Targeted Syntactic Evaluation of Language Models on Georgian Case Alignmentcited by 0
  33. 2026Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Modelscited by 0
  34. 2026Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandiccited by 0
  35. 2026Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italiancited by 0
  36. 2026Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairscited by 0
  37. 2026On the Limits of Model Merging for Multilinguality in Pre-Trainingcited by 0
  38. 2026When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputscited by 0

50 most cited of 71 All 71 (JSON)