mlsanityData to 7 Oct 2026

Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs

arXiv 2502.14830 ↗First posted 20 Feb 2025cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

While large language models demonstrate remarkable capabilities at task-specific applications through fine-tuning, extending these benefits across diverse languages is essential for broad accessibility. However, effective cross-lingual transfer is hindered by LLM performance gaps across languages and the scarcity of fine-tuning data in many languages. Through analysis of LLM internal representations from over 1,000+ language pairs, we discover that middle layers exhibit the strongest potential for cross-lingual alignment. Building on this finding, we propose a middle-layer alignment objective integrated into task-specific training. Our experiments on slot filling, machine translation, and structured text generation show consistent improvements in cross-lingual transfer, especially to lower-resource languages. The method is robust to the choice of alignment languages and generalizes to languages unseen during alignment. Furthermore, we show that separately trained alignment modules can be merged with existing task-specific modules, improving cross-lingual capabilities without full re-training. Our code is publicly available (https://github.com/dannigt/mid-align).

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 60 of its 91 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

60 of 91 references.

References · 60

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  9. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2019How multilingual is Multilingual BERT?cited by 642
38 more
  1. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  3. 2017Word Translation Without Parallel Datacited by 480
  4. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  5. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  7. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  8. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  9. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  10. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  11. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  12. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  13. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 127
  14. 2021Contrastive Learning for Many-to-many Multilingual Neural Machine Translationcited by 108
  15. 2023SONAR: Sentence-Level Multimodal and Language-Agnostic Representationscited by 105
  16. 2018Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddingscited by 83
  17. 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
  18. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 69
  19. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  20. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  21. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  22. 2023Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capabilitycited by 62
  23. 2022Cross-modal Contrastive Learning for Speech Translationcited by 61
  24. 2019The Missing Ingredient in Zero-Shot Neural Machine Translationcited by 59
  25. 2020Explicit Alignment Objectives for Multilingual Bidirectional Encoderscited by 50
  26. 2024Probing the Emergence of Cross-lingual Alignment during LLM Trainingcited by 38
  27. 2023The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Modelscited by 37
  28. 2015Transfer Learning for Speech and Language Processingcited by 35
  29. 2024Understanding Cross-Lingual Alignment -- A Surveycited by 34
  30. 2019Improving Zero-shot Translation with Language-Independent Constraintscited by 33
  31. 2024Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domainscited by 30
  32. 2023Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarizationcited by 28
  33. 2023Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmarkcited by 26
  34. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  35. 2024Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Interventioncited by 24
  36. 2023Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solutioncited by 22
  37. 2023Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understandingcited by 21
  38. 2022T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translationcited by 16

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 17.

26 papers.

Sort:

By area:

  1. 2025Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMscited by 11
  2. 2025Conditions for Catastrophic Forgetting in Multilingual Translationcited by 10
  3. 2025A quantitative analysis of semantic information in deep representations of text and imagescited by 7
  4. 2025Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretrainingcited by 6
  5. 2025Punctuation and Predicates in Language Modelscited by 5
  6. 2025Rethinking Cross-lingual Gaps from a Statistical Viewpointcited by 4
  7. 2026LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?cited by 3
  8. 2026Post-Training Language Models for Crosslingual Consistencycited by 3
  9. 2025MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoningcited by 2
  10. 2026Beyond Static Personas: Situational Personality Steering for Large Language Modelscited by 2
  11. 2026CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?cited by 2
  12. 2025Leveraging Intermediate Representations of Time Series Foundation Models for Anomaly Detectioncited by 1
14 more
  1. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 1
  2. 2026Beyond Language: Format-Agnostic Reasoning Subspaces in Large Language Modelscited by 1
  3. 2026How Far Do Auto-Interpretation Labels Generalize: A Controlled Study Across Languages, Scripts, and Rewordingscited by 1
  4. 2025OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusioncited by 0
  5. 2026Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoderscited by 0
  6. 2026Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmarkcited by 0
  7. 2026Multilingual Knowledge Transfer under Data Constraints via Lexical Interventionscited by 0
  8. 2026HiMed: Incentivizing Hindi Reasoning in Medical LLMscited by 0
  9. 2026Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Modelscited by 0
  10. 2026Visual Instruction Tuning Aligns Modalities through Abstractioncited by 0
  11. 2026Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inferencecited by 0
  12. 2026Cross-Lingual Representation Alignment by Token-Level Optimal Transport in a Language-Agnostic Spacecited by 0
  13. 2026Online Language Adaptive Sampling for Better Distributed Cross-lingual Gainscited by 0
  14. 2026Cross-Lingual Alignment for Decoder-Only Models using MoE Routerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers
  2. 2023Extrapolating Large Language Models to Non-English by Aligning Languages
  3. 2026Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
  4. 2025AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
  5. 2024Understanding Layer Significance in LLM Alignment
  6. 2022Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment
  7. 2024Concept Space Alignment in Multilingual LLMs
  8. 2025How Language Directions Align with Token Geometry in Multilingual LLMs