mlsanityData to 14 Oct 2026

Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models

arXiv 2411.07474 ↗First posted 12 Nov 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Language models (LMs) are capable of acquiring elements of human-like syntactic knowledge. Targeted syntactic evaluation tests have been employed to measure how well they form generalizations about syntactic phenomena in high-resource languages such as English. However, we still lack a thorough understanding of LMs' capacity for syntactic generalizations in low-resource languages, which are responsible for much of the diversity of syntactic patterns worldwide. In this study, we develop targeted syntactic evaluation tests for three low-resource languages (Basque, Hindi, and Swahili) and use them to evaluate five families of open-access multilingual Transformer LMs. We find that some syntactic tasks prove relatively easy for LMs while others (agreement in sentences containing indirect objects in Basque, agreement across a prepositional phrase in Swahili) are challenging. We additionally uncover issues with publicly available Transformers, including a bias toward the habitual aspect in Hindi in multilingual BERT and underperformance compared to similar-sized models in XGLM-4.5B.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

30 of 43 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2018Neural Network Acceptability Judgmentscited by 958
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  8. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  9. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  10. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  11. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  12. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
18 more
  1. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  2. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  3. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  4. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  5. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  6. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  7. 2020Intrinsic Probing through Dimension Selectioncited by 45
  8. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  9. 2018RNNs as psycholinguistic subjects: Syntactic state and grammatical dependencycited by 23
  10. 2021Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpuscited by 18
  11. 2018Can LSTM Learn to Capture Agreement? The Case of Basquecited by 17
  12. 2021DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharingcited by 16
  13. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 14
  14. 2023JCoLA: Japanese Corpus of Linguistic Acceptabilitycited by 11
  15. 2023NoCoLA: The Norwegian Corpus of Linguistic Acceptabilitycited by 10
  16. 2021Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Modelscited by 8
  17. 2021Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Modelscited by 5
  18. 2020Evaluating German Transformer Language Models with Syntactic Agreement Testscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 2.

10 papers.

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  2. 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
  3. 2025Linguistic Generalizations are not Rules: Impacts on Evaluation of LMscited by 10
  4. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 10
  5. 2025TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairscited by 9
  6. 2024Overview of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025)cited by 2
  7. 2025UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urducited by 2
  8. 2025Scaling and context steer LLMs along the same computational path as the human braincited by 1
  9. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  10. 2026Targeted Syntactic Evaluation of Language Models on Georgian Case Alignmentcited by 0