mlsanityData to 13 Oct 2026

CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMs

arXiv 2502.12476 ↗First posted 18 Feb 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Multilingual Large Language Models (LLMs) develop cross-lingual abilities despite being trained on limited parallel data. However, they often struggle to generate responses in the intended language, favoring high-resource languages such as English. In this work, we introduce CoCo-CoLa (Correct Concept - Correct Language), a novel metric to evaluate language adherence in multilingual LLMs. Using fine-tuning experiments on a closed-book QA task across seven languages, we analyze how training in one language affects others' performance. Our findings reveal that multilingual models share task knowledge across languages but exhibit biases in the selection of output language. We identify language-specific layers, showing that final layers play a crucial role in determining output language. Accordingly, we propose a partial training strategy that selectively fine-tunes key layers, improving language adherence while significantly reducing computational cost. Our method achieves comparable or superior performance to full fine-tuning, particularly for low-resource languages, offering a more efficient multilingual adaptation.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

39 of 50 references.

References · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,486
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  3. 2023Mistral 7Bcited by 3,123
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,978
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,908
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,323
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 750
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  10. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 280
  11. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 236
  12. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 229
27 more
  1. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 150
  2. 2024How do Large Language Models Handle Multilingualism?cited by 150
  3. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 116
  4. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  5. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 96
  6. 2023Do Multilingual Language Models Think Better in English?cited by 87
  7. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 72
  8. 2024Multilingual Instruction Tuning With Just a Pinch of Multilingualitycited by 71
  9. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  10. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  11. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  12. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 43
  13. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  14. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 38
  15. 2024AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Mergingcited by 34
  16. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  17. 2023Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfercited by 22
  18. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  19. 2023Hyperpolyglot LLMs: Cross-Lingual Interpretability in Token Embeddingscited by 19
  20. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformerscited by 18
  21. 2023Cross-Lingual Transfer with Target Language-Ready Task Adapterscited by 15
  22. 2024Exploring Multilingual Probing in Large Language Models: A Cross-Language Analysiscited by 12
  23. 2024Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learnerscited by 11
  24. 2024Zero-shot cross-lingual transfer in instruction tuning of large language modelscited by 10
  25. 2024Mechanistic interpretability of large language models with applications to the financial services industrycited by 5
  26. 2024Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inferencecited by 3
  27. 2024No Optimal Language Set Exists for Multilingual Instruction Tuning: Insights from a Linguistically-Informed Studycited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026How Open Must Language Models be to Enable Reliable Scientific Inference?cited by 2
  2. 2026Linear Semantic Segmentation for Low-Resource Spoken Dialectscited by 1
  3. 2026Disentangling Language Roles in Multilingual LLM Task Executioncited by 1
  4. 2026What Does Post-Training Change in Multilingual Reasoning?cited by 0