What if I ask in textitalia lingua? Measuring Functional Similarity Across Languages
Not on the map. Why?
Abstract (arXiv)
How similar are model outputs across languages? In this work, we study this question using a recently proposed model similarity metric kappap applied to 20 languages and 47 subjects in GlobalMMLU. Our analysis reveals that a model's responses become increasingly consistent across languages as its size and capability grow. Interestingly, models exhibit greater cross-lingual consistency within themselves than agreement with other models prompted in the same language. These results highlight not only the value of kappap as a practical tool for evaluating multilingual reliability, but also its potential to guide the development of more consistent multilingual systems.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 12 of its 16 references that are in the map. First posted on arXiv 4 Sep 2025.
Builds on
12 of 16 references.
References · 12
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
- 2024The Platonic Representation Hypothesiscited by 362
- 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 128
- 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
- 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
- 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
- 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
- 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 45
- 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
- 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 34
- 2024Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Modelscited by 17
- 2024Decoding Multilingual Moral Preferences: Unveiling LLM's Biases Through the Moral Machine Experimentcited by 11
Built on by
Citing papers per year: 2026: 1.
1 paper.