mlsanityData to 14 Oct 2026

Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?

arXiv 2507.12838 ↗First posted 17 Jul 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain the factuality of the model knowledge across languages, and preserve the parity of language model performance. We are thus interested in analyzing, evaluating, and interpreting cross-lingual consistency for factual knowledge. To facilitate our study, we examine multiple pretrained models and tuned models with code-mixed coreferential statements that convey identical knowledge across languages. Interpretability approaches are leveraged to analyze the behavior of a model in cross-lingual contexts, showing different levels of consistency in multilingual models, subject to language families, linguistic factors, scripts, and a bottleneck in cross-lingual consistency on a particular layer. Code-switching training and cross-lingual word alignment objectives show the most promising results, emphasizing the worthiness of cross-lingual alignment supervision and code-switching strategies for both multilingual performance and cross-lingual consistency enhancement. In addition, experimental results suggest promising result for calibrating consistency in the test time via activation patching.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 17 Jul 2025.

Builds on

40 of 57 references.

References · 40

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  4. 2019Language Models as Knowledge Bases?cited by 1,936
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2022Toy Models of Superpositioncited by 971
  9. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  10. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  11. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  12. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
28 more
  1. 2021Causal Abstractions of Neural Networkscited by 422
  2. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  3. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
  4. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  5. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  6. 2024How do Large Language Models Handle Multilingualism?cited by 150
  7. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  8. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  9. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  10. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  11. 2021X-FACT: A New Benchmark Dataset for Multilingual Fact Checkingcited by 91
  12. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  13. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  14. 2020PMI-Masking: Principled masking of correlated spanscited by 54
  15. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 44
  16. 2021Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignmentcited by 41
  17. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 39
  18. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  19. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  20. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  21. 2024mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?cited by 21
  22. 2024Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translationcited by 19
  23. 2024Cross-lingual Editing in Multilingual Language Modelscited by 18
  24. 2024Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledgecited by 14
  25. 2023Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networkscited by 13
  26. 2024Exploring Alignment in Shared Cross-lingual Spacescited by 13
  27. 2022EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switchingcited by 8
  28. 2023Polyglot or Not? Measuring Multilingual Encyclopedic Knowledge in Foundation Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Discursive Circuits: How Do Language Models Understand Discourse Relations?cited by 4
  2. 2026Post-Training Language Models for Crosslingual Consistencycited by 3
  3. 2026When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Modelscited by 1
  4. 2026Learning to Route Languages for Multilingual Policy Optimizationcited by 1
  5. 2026Adversarial Concept Search: Predicting Compositional Errors From Feature Geometrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Entity-Level Alignment in Crosslingual Consistency
  2. 2024Evaluating Knowledge-based Cross-lingual Inconsistency in Large Language Models
  3. 2025Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs
  4. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
  5. 2025Cross-linguistic disagreement as a conflict of semantic alignment norms in multilingual AI~Linguistic Diversity as a Problem for Philosophy, Cognitive Science, and AI~
  6. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers
  7. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment
  8. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple senses