mlsanityData to 7 Oct 2026

RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs

arXiv 2406.19232 ↗First posted 27 Jun 2024cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Minimal pairs are a well-established approach to evaluating the grammatical knowledge of language models. However, existing resources for minimal pairs address a limited number of languages and lack diversity of language-specific grammatical phenomena. This paper introduces the Russian Benchmark of Linguistic Minimal Pairs (RuBLiMP), which includes 45k pairs of sentences that differ in grammaticality and isolate a morphological, syntactic, or semantic phenomenon. In contrast to existing benchmarks of linguistic minimal pairs, RuBLiMP is created by applying linguistic perturbations to automatically annotated sentences from open text corpora and carefully curating test data. We describe the data collection protocol and present the results of evaluating 25 language models in various scenarios. We find that the widely used language models for Russian are sensitive to morphological and agreement-oriented contrasts but fall behind humans on phenomena requiring understanding of structural relations, negation, transitivity, and tense. RuBLiMP, the codebase, and other materials are publicly available.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 40 of its 77 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

40 of 77 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2023Mistral 7Bcited by 3,065
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  12. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
28 more
  1. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  2. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  3. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  4. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  5. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  6. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  7. 2020Syntactic Structure from Deep Learningcited by 99
  8. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  9. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  10. 2019Joint Reasoning for Temporal and Causal Relationscited by 60
  11. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  12. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 54
  13. 2023Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformationcited by 51
  14. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 44
  15. 2019Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmarkcited by 32
  16. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  17. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  18. 2015Morphological Analyzer and Generator for Russian and Ukrainian Languagescited by 25
  19. 2021Learning from Crowds with Crowd-Kitcited by 23
  20. 2019LINSPECTOR: Multilingual Probing Tasks for Word Representationscited by 22
  21. 2021DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharingcited by 16
  22. 2022SLING: Sino Linguistic Evaluation of Large Language Modelscited by 15
  23. 2019Probing Multilingual Sentence Representations With X-Probecited by 13
  24. 2023NoCoLA: The Norwegian Corpus of Linguistic Acceptabilitycited by 9
  25. 2021Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Modelscited by 7
  26. 2017Human and Machine Judgements for Russian Semantic Relatednesscited by 4
  27. 2021RuSentEval: Linguistic Source, Encoder Force!cited by 4
  28. 2022Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 6.

23 papers.

Sort:

By area:

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  2. 2024A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinesecited by 13
  3. 2025XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairscited by 9
  4. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  5. 2025Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluationscited by 7
  6. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 7
  7. 2025Minimal Pair-Based Evaluation of Code-Switchingcited by 6
  8. 2025MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMscited by 4
  9. 2025Explain-then-Process: Using Grammar Prompting to Enhance Grammatical Acceptability Judgmentscited by 2
  10. 2025UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urducited by 2
  11. 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
  12. 2024Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Modelscited by 1
11 more
  1. 2024Linguistically Grounded Analysis of Language Models using Shapley Head Valuescited by 1
  2. 2025RusCode: Russian Cultural Code Benchmark for Text-to-Image Generationcited by 1
  3. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  4. 2025Detecting Spelling and Grammatical Anomalies in Russian Poetry Textscited by 0
  5. 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibilitycited by 0
  6. 2025Eye of Judgement: Dissecting the Evaluation of Russian-speaking LLMs with POLLUXcited by 0
  7. 2026Targeted Syntactic Evaluation of Language Models on Georgian Case Alignmentcited by 0
  8. 2026Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairscited by 0
  9. 2026When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputscited by 0
  10. 2026Linear representations of grammaticality in neural language modelscited by 0
  11. 2026Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Modelscited by 0