mlsanityData to 14 Oct 2026

TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs

arXiv 2506.13487 ↗First posted 16 Jun 2025cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce TurBLiMP, the first Turkish benchmark of linguistic minimal pairs, designed to evaluate the linguistic abilities of monolingual and multilingual language models (LMs). Covering 16 linguistic phenomena with 1000 minimal pairs each, TurBLiMP fills an important gap in linguistic evaluation resources for Turkish. In designing the benchmark, we give extra attention to two properties of Turkish that remain understudied in current syntactic evaluations of LMs, namely word order flexibility and subordination through morphological processes. Our experiments on a wide range of LMs and a newly collected set of human acceptability judgments reveal that even cutting-edge Large LMs still struggle with grammatical phenomena that are not challenging for humans, and may also exhibit different sensitivities to word order and morphological complexity compared to humans.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 25 of its 55 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

25 of 55 references.

References · 25

  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  2. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  3. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  4. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  5. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  6. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  7. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  8. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  9. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  10. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  11. 2024Language models align with human judgments on key grammatical constructionscited by 49
  12. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
13 more
  1. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
  2. 2018Do Language Models Understand Anything? On the Ability of LSTMs to Understand Negative Polarity Itemscited by 36
  3. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 30
  4. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  5. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  6. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  7. 2017Linguistically Motivated Vocabulary Reduction for Neural Machine Translation from Turkish to Englishcited by 23
  8. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 19
  9. 2022SLING: Sino Linguistic Evaluation of Large Language Modelscited by 15
  10. 2023Evaluating Neural Language Models as Cognitive Models of Language Acquisitioncited by 15
  11. 2024Controlled Evaluation of Syntactic Knowledge in Multilingual Language Modelscited by 10
  12. 2024Introducing cosmosGPT: Monolingual Training for Turkish Language Modelscited by 8
  13. 2020Resources for Turkish Dependency Parsing: Introducing the BOUN Treebank and the BoAT Annotation Toolcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 2.

9 papers.

  1. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 10
  2. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 8
  3. 2025Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluationscited by 7
  4. 2025Tokens with Meaning: A Hybrid Tokenization Approach for Turkishcited by 7
  5. 2025SindBERT, the Sailor: Charting the Seas of Turkish NLPcited by 4
  6. 2025UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urducited by 2
  7. 2025Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Settingcited by 2
  8. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  9. 2026ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratchcited by 2