mlsanityData to 14 Oct 2026

XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs

arXiv 2502.19737 ↗First posted 27 Feb 2025cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

We introduce XCOMPS in this work, a multilingual conceptual minimal pair dataset covering 17 languages. Using this dataset, we evaluate LLMs' multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. By comparing base, instruction-tuned, and knowledge-distilled models, we find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) Instruction tuning improves performance in concept understanding but does not enhance internal competence; knowledge distillation can enhance internal competence in conceptual understanding for low-resource languages with limited gains in explicit task performance. 4) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 31 of its 55 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

31 of 55 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  12. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
19 more
  1. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  2. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  3. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  4. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  5. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  6. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  7. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  8. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  9. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  10. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  11. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  12. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  13. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 46
  14. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 45
  15. 2024Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairscited by 24
  16. 2024RuBLiMP: Russian Benchmark of Linguistic Minimal Pairscited by 24
  17. 2021DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharingcited by 16
  18. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  19. 2024BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?cited by 11
  2. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 10
  3. 2025Large Language Model probabilities cannot distinguish between possible and impossible languagecited by 5
  4. 2026From Chains to DAGs: Probing the Graph Structure of Reasoning in LLMscited by 2
  5. 2026Structural priors for data-efficient language learningcited by 1
  6. 2026The First ChineseBabyLM Challenge: training data-efficient and cognitively plausible language models for Chinesecited by 0
  7. 2026Grammatical "grandmother neurons" are rare in LLMscited by 0
  8. 2026Feeding BabyLMs Macaroni: Code-Switching Curricula Cause Cross-Lingual Convergencecited by 0
  9. 2026Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AIcited by 0
  10. 2026Introducing Code-Switched Contexts to Cognitively-Inspired Bilingual Model Trainingcited by 0
  11. 2026Language Exposure Matters: Distribution and Order in Multilingual BabyLM Pre-trainingcited by 0
  12. 2026Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses
  2. 2026Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs
  3. 2023Large Linguistic Models: Investigating LLMs' metalinguistic abilities
  4. 2025The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang
  5. 2026Rethinking the Multilingual Reasoning Gap with Layer Swap
  6. 2025When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
  7. 2024xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning
  8. 2024Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?