# CoCo-CoLa: Evaluating and Improving Language Adherence in Multilingual LLMs

arXiv: https://arxiv.org/abs/2502.12476 · HTML: https://mlsanity.com/paper/2502.12476/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 50 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 30 of 39)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [How do Large Language Models Handle Multilingualism?](https://mlsanity.com/paper/2402.18815/) (2024)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics](https://mlsanity.com/paper/2410.21272/) (2024)
- [Not All Layers of LLMs Are Necessary During Inference](https://mlsanity.com/paper/2403.02181/) (2024)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [Multilingual Instruction Tuning With Just a Pinch of Multilinguality](https://mlsanity.com/paper/2401.01854/) (2024)
- [Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering](https://mlsanity.com/paper/2210.01613/) (2022)
- [Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?](https://mlsanity.com/paper/2106.01561/) (2021)
- [Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts](https://mlsanity.com/paper/2306.11372/) (2023)
- [Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers](https://mlsanity.com/paper/2411.08745/) (2024)
- [On Large Language Models' Hallucination with Regard to Known Facts](https://mlsanity.com/paper/2403.20009/) (2024)
- [Pre-trained Large Language Models Use Fourier Features to Compute Addition](https://mlsanity.com/paper/2406.03445/) (2024)
- [Probing the Emergence of Cross-lingual Alignment during LLM Training](https://mlsanity.com/paper/2406.13229/) (2024)
- [AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging](https://mlsanity.com/paper/2402.18913/) (2024)
- [Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly](https://mlsanity.com/paper/2404.04659/) (2024)
- [Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer](https://mlsanity.com/paper/2305.14280/) (2023)
- [Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models](https://mlsanity.com/paper/2410.01335/) (2024)

## Built on by (most cited first, 4 of 4)

- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [Linear Semantic Segmentation for Low-Resource Spoken Dialects](https://mlsanity.com/paper/2605.06276/) (2026)
- [Disentangling Language Roles in Multilingual LLM Task Execution](https://mlsanity.com/paper/2605.27649/) (2026)
- [What Does Post-Training Change in Multilingual Reasoning?](https://mlsanity.com/paper/2609.37104/) (2026)

## Abstract (arXiv)

Multilingual Large Language Models (LLMs) develop cross-lingual abilities despite being trained on limited parallel data. However, they often struggle to generate responses in the intended language, favoring high-resource languages such as English. In this work, we introduce CoCo-CoLa (Correct Concept - Correct Language), a novel metric to evaluate language adherence in multilingual LLMs. Using fine-tuning experiments on a closed-book QA task across seven languages, we analyze how training in one language affects others' performance. Our findings reveal that multilingual models share task knowledge across languages but exhibit biases in the selection of output language. We identify language-specific layers, showing that final layers play a crucial role in determining output language. Accordingly, we propose a partial training strategy that selectively fine-tunes key layers, improving language adherence while significantly reducing computational cost. Our method achieves comparable or superior performance to full fine-tuning, particularly for low-resource languages, offering a more efficient multilingual adaptation.