# LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy

arXiv: https://arxiv.org/abs/2502.11405 · HTML: https://mlsanity.com/paper/2502.11405/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 13 of 13)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Orca-Math: Unlocking the potential of SLMs in Grade School Math](https://mlsanity.com/paper/2402.14830/) (2024)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations](https://mlsanity.com/paper/2310.20246/) (2023)
- [LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages](https://mlsanity.com/paper/2407.05975/) (2024)
- [EVLM: An Efficient Vision-Language Model for Visual Understanding](https://mlsanity.com/paper/2407.14177/) (2024)
- [MindMerger: Efficient Boosting LLM Reasoning in non-English Languages](https://mlsanity.com/paper/2405.17386/) (2024)
- [ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI](https://mlsanity.com/paper/2401.15471/) (2024)
- [Rethinking and Improving Natural Language Generation with Layer-Wise Multi-View Decoding](https://mlsanity.com/paper/2005.08081/) (2020)

## Built on by (most cited first, 10 of 10)

- [Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering](https://mlsanity.com/paper/2601.14050/) (2026)
- [Less Data Less Tokens: Multilingual Unification Learning for Efficient Test-Time Reasoning in LLMs](https://mlsanity.com/paper/2506.18341/) (2025)
- [MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning](https://mlsanity.com/paper/2509.08105/) (2025)
- [Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality](https://mlsanity.com/paper/2603.17512/) (2026)
- [SAT: Balancing Reasoning Accuracy and Efficiency with Stepwise Adaptive Thinking](https://mlsanity.com/paper/2604.07922/) (2026)
- [GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models](https://mlsanity.com/paper/2605.01256/) (2026)
- [Single-Pass, Depth-Selective Reading for Multi-Aspect Sentiment Analysis](https://mlsanity.com/paper/2605.20998/) (2026)
- [Enhancing Multilingual Reasoning via Steerable Model Merging](https://mlsanity.com/paper/2606.19002/) (2026)
- [VFA: Empowering Multilingual MLLMs via Vision-Free Adaptation](https://mlsanity.com/paper/2608.26155/) (2026)
- [Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning](https://mlsanity.com/paper/2609.10445/) (2026)

## Abstract (arXiv)

Despite being pretrained on multilingual corpora, large language models (LLMs) exhibit suboptimal performance on low-resource languages. Recent approaches have leveraged multilingual encoders alongside LLMs by introducing trainable parameters connecting the two models. However, these methods typically focus on the encoder's output, overlooking valuable information from other layers. We propose aname (mname), a framework that integrates representations from all encoder layers, coupled with the attaname mechanism to enable layer-wise interaction between the LLM and the multilingual encoder. Extensive experiments on multilingual reasoning tasks, along with analyses of learned representations, show that our approach consistently outperforms existing baselines.