mlsanityData to 7 Oct 2026

Enhancing Multilingual Reasoning via Steerable Model Merging

arXiv 2606.19002 ↗First posted 17 Jun 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Model merging is an effective technique for composing the capabilities of a multilingual model and a reasoning model. It has achieved promising generalization in multilingual reasoning tasks by aligning feature spaces of different models. However, the merged single model often fails to address the conflicts between source models, leading to suboptimal performance. In other words, the one-size-fits-all merging strategy may not align with the characteristics of different inputs which may require prioritizing certain models over others. To this end, we propose a Steerable Model Merging (ST-Merge) framework to modulate the contribution of each source model. To realize this idea, we introduce a gated cross-attention mechanism to weight or filter the two attended source models in an adaptive manner. Extensive experiments demonstrate that ST-Merge consistently outperforms multiple strong baselines on four multilingual reasoning benchmarks across 21 different languages.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 14 of its 24 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

14 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy17 Feb
  2. 2024Question Translation Training for Better Multilingual Reasoning15 Jan

Other references · 12

  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  4. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  5. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  6. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  7. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  8. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  9. 2025AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimizationcited by 25
  10. 2022Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translationcited by 21
  11. 2025CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detectioncited by 6
  12. 2021An Interpretable Framework for Drug-Target Interaction with Gated Cross Attentioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging
  2. 2025Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging
  3. 2024Unconstrained Model Merging for Enhanced LLM Reasoning
  4. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?
  5. 2025A Survey of Multilingual Reasoning in Language Models
  6. 2025SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment
  7. 2026AdaMame: A Training Recipe for Adaptive Multilingual Reasoning
  8. 2023Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention