# MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusion

arXiv: https://arxiv.org/abs/2503.16212 · HTML: https://mlsanity.com/paper/2503.16212/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 35)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2310.03731/) (2023)
- [MathScale: Scaling Instruction Tuning for Mathematical Reasoning](https://mlsanity.com/paper/2403.02884/) (2024)
- [Augmenting Data with Mixup for Sentence Classification: An Empirical Study](https://mlsanity.com/paper/1905.08941/) (2019)
- [InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning](https://mlsanity.com/paper/2402.06332/) (2024)
- [Common 7B Language Models Already Possess Strong Math Capabilities](https://mlsanity.com/paper/2403.04706/) (2024)
- [RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold](https://mlsanity.com/paper/2406.14532/) (2024)
- [Sequence-Level Mixed Sample Data Augmentation](https://mlsanity.com/paper/2011.09039/) (2020)
- [Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision](https://mlsanity.com/paper/2411.16579/) (2024)
- [Augmenting Math Word Problems via Iterative Question Composing](https://mlsanity.com/paper/2401.09003/) (2024)
- [Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning](https://mlsanity.com/paper/2403.02333/) (2024)
- [SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup](https://mlsanity.com/paper/2010.02322/) (2020)
- [MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning](https://mlsanity.com/paper/2310.05506/) (2023)
- [Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning](https://mlsanity.com/paper/2407.00782/) (2024)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning](https://mlsanity.com/paper/2408.14774/) (2024)

## Built on by (most cited first, 30 of 30)

- [SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.08989/) (2025)
- [AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery](https://mlsanity.com/paper/2604.05550/) (2026)
- [A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law](https://mlsanity.com/paper/2505.02665/) (2025)
- [Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2507.17512/) (2025)
- [OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value](https://mlsanity.com/paper/2512.14051/) (2025)
- [A Survey on Collaborative Mechanisms Between Large and Small Language Models](https://mlsanity.com/paper/2505.07460/) (2025)
- [REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once](https://mlsanity.com/paper/2507.10541/) (2025)
- [Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning](https://mlsanity.com/paper/2510.04081/) (2025)
- [IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment](https://mlsanity.com/paper/2505.12762/) (2025)
- [AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generation](https://mlsanity.com/paper/2510.19361/) (2025)
- [DAMR: Efficient and Adaptive Context-Aware Knowledge Graph Question Answering with LLM-Guided MCTS](https://mlsanity.com/paper/2508.00719/) (2025)
- [ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoning](https://mlsanity.com/paper/2509.21070/) (2025)
- [Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPO](https://mlsanity.com/paper/2603.03314/) (2026)
- [SIGMA: Refining Large Language Model Reasoning via Sibling-Guided Monte Carlo Augmentation](https://mlsanity.com/paper/2506.06470/) (2025)
- [Closing the Data Loop: Using OpenDataArena to Engineer Superior Training Datasets](https://mlsanity.com/paper/2601.09733/) (2025)
- [ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure](https://mlsanity.com/paper/2602.01472/) (2026)
- [Learning from Contrasts: Synthesizing Reasoning Paths from Diverse Search Trajectories](https://mlsanity.com/paper/2604.11365/) (2026)
- [Beyond Rejection Sampling: Trajectory Fusion for Scaling Mathematical Reasoning](https://mlsanity.com/paper/2602.04391/) (2026)
- [Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning](https://mlsanity.com/paper/2505.14684/) (2025)
- [Tracing the Roots: A Multi-Agent Framework for Uncovering Data Lineage in Post-Training LLMs](https://mlsanity.com/paper/2604.10480/) (2026)
- [UniCode: Augmenting Evaluation for Code Reasoning](https://mlsanity.com/paper/2510.17868/) (2025)
- [MathMixup: Boosting LLM Mathematical Reasoning with Difficulty-Controllable Data Synthesis and Curriculum Learning](https://mlsanity.com/paper/2601.17006/) (2026)
- [Lossless Anti-Distillation Sampling](https://mlsanity.com/paper/2605.18829/) (2026)
- [Bidirectional Curriculum Generation: A Multi-Agent Framework for Data-Efficient Mathematical Reasoning](https://mlsanity.com/paper/2603.05120/) (2026)
- [Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs](https://mlsanity.com/paper/2604.02230/) (2026)
- [D2Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2605.17037/) (2026)
- [Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models](https://mlsanity.com/paper/2606.03624/) (2026)
- [ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management](https://mlsanity.com/paper/2607.20764/) (2026)
- [SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature](https://mlsanity.com/paper/2608.30214/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown impressive progress in mathematical reasoning. While data augmentation is promising to enhance mathematical problem-solving ability, current approaches are predominantly limited to instance-level modifications-such as rephrasing or generating syntactic variations-which fail to capture and leverage the intrinsic relational structures inherent in mathematical knowledge. Inspired by human learning processes, where mathematical proficiency develops through systematic exposure to interconnected concepts, we introduce MathFusion, a novel framework that enhances mathematical reasoning through cross-problem instruction synthesis. MathFusion implements this through three fusion strategies: (1) sequential fusion, which chains related problems to model solution dependencies; (2) parallel fusion, which combines analogous problems to reinforce conceptual understanding; and (3) conditional fusion, which creates context-aware selective problems to enhance reasoning flexibility. By applying these strategies, we generate a new dataset, MathFusionQA, followed by fine-tuning models (DeepSeekMath-7B, Mistral-7B, Llama3-8B) on it. Experimental results demonstrate that MathFusion achieves substantial improvements in mathematical reasoning while maintaining high data efficiency, boosting performance by 18.0 points in accuracy across diverse benchmarks while requiring only 45K additional synthetic instructions, representing a substantial improvement over traditional single-instruction approaches. Our datasets, models, and code are publicly available at https://github.com/QizhiPei/mathfusion.