mlsanityData to 14 Oct 2026

Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features

arXiv 2307.05454 ↗First posted 11 Jul 2023cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

A challenge towards developing NLP systems for the world's languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologically-aware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models' behavior in light of specific linguistic features in 12 typologically diverse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish. Our findings motivate the development of models that address these blind spots.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

35 of 57 references.

References · 35

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
23 more
  1. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  2. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  3. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  4. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  5. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  6. 2018Stress Test Evaluation for Natural Language Inferencecited by 213
  7. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  8. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  9. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  10. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  11. 2021Language Model Evaluation Beyond Perplexitycited by 78
  12. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  13. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  14. 2022Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Gocited by 47
  15. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  16. 2019Pushing the Limits of Low-Resource Morphological Inflectioncited by 29
  17. 2020TaxiNLI: Taking a Ride up the NLU Hillcited by 27
  18. 2019How Large Are Lions? Inducing Distributions over Quantitative Attributescited by 26
  19. 2022LMentry: A Language Model Benchmark of Elementary Language Taskscited by 22
  20. 2018Can LSTM Learn to Capture Agreement? The Case of Basquecited by 17
  21. 2021Minimax and Neyman-Pearson Meta-Learning for Outlier Languagescited by 10
  22. 2021Analyzing the Effects of Reasoning Types on Cross-Lingual Transfer Performancecited by 6
  23. 2022Multilingual CheckList: Generation and Evaluationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 1.

6 papers.

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  2. 2024Probing the Category of Verbal Aspect in Transformer Language Modelscited by 5
  3. 2025IMPACT: Inflectional Morphology Probes Across Complex Typologiescited by 4
  4. 2025Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+cited by 3
  5. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  6. 2025AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Modelscited by 0