# MindMerger: Efficient Boosting LLM Reasoning in non-English Languages

arXiv: https://arxiv.org/abs/2405.17386 · HTML: https://mlsanity.com/paper/2405.17386/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 27 May 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Scaling Relationship on Learning Mathematical Reasoning with Large Language Models](https://mlsanity.com/paper/2308.01825/) (2023)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge](https://mlsanity.com/paper/2102.03315/) (2021)
- [Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations](https://mlsanity.com/paper/2310.20246/) (2023)
- [xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning](https://mlsanity.com/paper/2401.07037/) (2024)
- [Question Translation Training for Better Multilingual Reasoning](https://mlsanity.com/paper/2401.07817/) (2024)
- [Forward-Backward Reasoning in Large Language Models for Mathematical Verification](https://mlsanity.com/paper/2308.07758/) (2023)
- [LLM Augmented LLMs: Expanding Capabilities through Composition](https://mlsanity.com/paper/2401.02412/) (2024)
- [LangBridge: Multilingual Reasoning Without Multilingual Supervision](https://mlsanity.com/paper/2401.10695/) (2024)

## Built on by (most cited first, 21 of 21)

- [LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages](https://mlsanity.com/paper/2407.05975/) (2024)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [A Survey of Multilingual Reasoning in Language Models](https://mlsanity.com/paper/2502.09457/) (2025)
- [Data Contamination Can Cross Language Barriers](https://mlsanity.com/paper/2406.13236/) (2024)
- [Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks](https://mlsanity.com/paper/2410.11005/) (2024)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging](https://mlsanity.com/paper/2406.16330/) (2024)
- [SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment](https://mlsanity.com/paper/2501.03681/) (2025)
- [Improving Multilingual Language Models by Aligning Representations through Steering](https://mlsanity.com/paper/2505.12584/) (2025)
- [Unveiling Over-Memorization in Finetuning LLMs for Reasoning Tasks](https://mlsanity.com/paper/2508.04117/) (2025)
- [LayAlign: Enhancing Multilingual Reasoning in Large Language Models via Layer-Wise Adaptive Fusion and Alignment Strategy](https://mlsanity.com/paper/2502.11405/) (2025)
- [Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only](https://mlsanity.com/paper/2410.11055/) (2024)
- [Lens: Rethinking Multilingual Enhancement for Large Language Models](https://mlsanity.com/paper/2410.04407/) (2024)
- [Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning](https://mlsanity.com/paper/2601.18722/) (2026)
- [Language Specific Knowledge: Do Models Know Better in X than in English?](https://mlsanity.com/paper/2505.14990/) (2025)
- [Cross-lingual Self-Consistency for Multilingual Reasoning with Language Models](https://mlsanity.com/paper/2606.01464/) (2026)
- [MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning](https://mlsanity.com/paper/2509.08105/) (2025)
- [Extracting and Combining Abilities For Building Multi-lingual Ability-enhanced Large Language Models](https://mlsanity.com/paper/2410.07825/) (2024)
- [Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages](https://mlsanity.com/paper/2508.09091/) (2025)
- [OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulation](https://mlsanity.com/paper/2510.03771/) (2025)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)

## Abstract (arXiv)

Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially in low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively.