# Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs

arXiv: https://arxiv.org/abs/2502.14830 · HTML: https://mlsanity.com/paper/2502.14830/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 91 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 30 of 60)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Exploiting Similarities among Languages for Machine Translation](https://mlsanity.com/paper/1309.4168/) (2013)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Composable Sparse Fine-Tuning for Cross-Lingual Transfer](https://mlsanity.com/paper/2110.07560/) (2021)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://mlsanity.com/paper/2105.09501/) (2021)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder](https://mlsanity.com/paper/1611.04798/) (2016)
- [Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://mlsanity.com/paper/1902.09492/) (2019)

## Built on by (most cited first, 29 of 29)

- [Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs](https://mlsanity.com/paper/2504.09378/) (2025)
- [Conditions for Catastrophic Forgetting in Multilingual Translation](https://mlsanity.com/paper/2510.19546/) (2025)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining](https://mlsanity.com/paper/2509.05291/) (2025)
- [Punctuation and Predicates in Language Models](https://mlsanity.com/paper/2508.14067/) (2025)
- [Rethinking Cross-lingual Gaps from a Statistical Viewpoint](https://mlsanity.com/paper/2510.15551/) (2025)
- [LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?](https://mlsanity.com/paper/2601.20009/) (2026)
- [Post-Training Language Models for Crosslingual Consistency](https://mlsanity.com/paper/2603.04678/) (2026)
- [MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning](https://mlsanity.com/paper/2509.08105/) (2025)
- [Multimodal In-context Learning for ASR of Low-resource Languages](https://mlsanity.com/paper/2601.05707/) (2026)
- [Layer-Targeted Multilingual Knowledge Erasure in Large Language Models](https://mlsanity.com/paper/2602.22562/) (2026)
- [Beyond Static Personas: Situational Personality Steering for Large Language Models](https://mlsanity.com/paper/2604.13846/) (2026)
- [CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?](https://mlsanity.com/paper/2607.04029/) (2026)
- [Leveraging Intermediate Representations of Time Series Foundation Models for Anomaly Detection](https://mlsanity.com/paper/2509.12650/) (2025)
- [Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Models](https://mlsanity.com/paper/2605.09496/) (2026)
- [How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordings](https://mlsanity.com/paper/2606.00356/) (2026)
- [OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion](https://mlsanity.com/paper/2512.00234/) (2025)
- [Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoders](https://mlsanity.com/paper/2603.19771/) (2026)
- [Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark](https://mlsanity.com/paper/2604.12744/) (2026)
- [Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions](https://mlsanity.com/paper/2605.23885/) (2026)
- [HiMed: Incentivizing Hindi Reasoning in Medical LLMs](https://mlsanity.com/paper/2605.24635/) (2026)
- [Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models](https://mlsanity.com/paper/2605.28306/) (2026)
- [Visual Instruction Tuning Aligns Modalities through Abstraction](https://mlsanity.com/paper/2606.03871/) (2026)
- [Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference](https://mlsanity.com/paper/2608.04904/) (2026)
- [Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Space](https://mlsanity.com/paper/2609.06381/) (2026)
- [Online Language Adaptive Sampling for Better Distributed Cross-lingual Gains](https://mlsanity.com/paper/2609.14969/) (2026)
- [Cross-Lingual Alignment for Decoder-Only Models using MoE Routers](https://mlsanity.com/paper/2610.01921/) (2026)
- [Predicting Steering Vectors and Adapter Weights for Few-Shot Author-Style Transfer](https://mlsanity.com/paper/2610.03163/) (2026)
- [Cross-Lingual Transfer of Evil Vectors Is Directionally Asymmetric](https://mlsanity.com/paper/2610.17880/) (2026)

## Abstract (arXiv)

While large language models demonstrate remarkable capabilities at task-specific applications through fine-tuning, extending these benefits across diverse languages is essential for broad accessibility. However, effective cross-lingual transfer is hindered by LLM performance gaps across languages and the scarcity of fine-tuning data in many languages. Through analysis of LLM internal representations from over 1,000+ language pairs, we discover that middle layers exhibit the strongest potential for cross-lingual alignment. Building on this finding, we propose a middle-layer alignment objective integrated into task-specific training. Our experiments on slot filling, machine translation, and structured text generation show consistent improvements in cross-lingual transfer, especially to lower-resource languages. The method is robust to the choice of alignment languages and generalizes to languages unseen during alignment. Furthermore, we show that separately trained alignment modules can be merged with existing task-specific modules, improving cross-lingual capabilities without full re-training. Our code is publicly available (https://github.com/dannigt/mid-align).