mlsanityData to 14 Oct 2026

MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning

arXiv 2310.03731 ↗First posted 5 Oct 2023cs.CL cs.AI cs.CVcited by 156

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

The recently released GPT-4 Code Interpreter has demonstrated remarkable proficiency in solving challenging math problems, primarily attributed to its ability to seamlessly reason with natural language, generate code, execute code, and continue reasoning based on the execution output. In this paper, we present a method to fine-tune open-source language models, enabling them to use code for modeling and deriving math equations and, consequently, enhancing their mathematical reasoning abilities. We propose a method of generating novel and high-quality datasets with math problems and their code-based solutions, referred to as MathCodeInstruct. Each solution interleaves natural language, code, and execution results. We also introduce a customized supervised fine-tuning and inference approach. This approach yields the MathCoder models, a family of models capable of generating code-based solutions for solving challenging math problems. Impressively, the MathCoder models achieve state-of-the-art scores among open-source LLMs on the MATH (45.2%) and GSM8K (83.9%) datasets, substantially outperforming other open-source alternatives. Notably, the MathCoder model not only surpasses ChatGPT-3.5 and PaLM-2 on GSM8K and MATH but also outperforms GPT-4 on the competition-level MATH dataset. The dataset and models will be released at https://github.com/mathllm/MathCoder.

Key facts. Cited by 156 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 53 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

43 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  2. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification15 Aug
  3. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Models3 Aug
  4. 2022Solving Quantitative Reasoning Problems with Language Models29 Jun

Other references · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
27 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2023PaLM 2 Technical Reportcited by 1,162
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2023LIMA: Less Is More for Alignmentcited by 1,100
  10. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  13. 2023StarCoder: may the source be with you!cited by 799
  14. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  15. 2022Galactica: A Large Language Model for Sciencecited by 744
  16. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  17. 2023Instruction Tuning with GPT-4cited by 686
  18. 2022PAL: Program-aided Language Modelscited by 652
  19. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  20. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  21. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  22. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 388
  23. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  24. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  25. 2023Self-Alignment with Instruction Backtranslationcited by 151
  26. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  27. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 55; 2025: 68; 2026: 31.

156 papers.

Sort:

By area:

  1. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  2. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  4. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  5. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  6. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 286
  7. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  9. 2025ToRL: Scaling Tool-Integrated RLcited by 169
  10. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  11. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  12. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
38 more
  1. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  2. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 119
  3. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  4. 2023Learning From Mistakes Makes LLM Better Reasonercited by 101
  5. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 99
  6. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  7. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  8. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  9. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  10. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  11. 2024Step-level Value Preference Optimization for Mathematical Reasoningcited by 72
  12. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 70
  13. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 70
  14. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
  15. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  16. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  17. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  18. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  19. 2026Code as Agent Harnesscited by 47
  20. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 45
  21. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  22. 2025MAM: Modular Multi-Agent Framework for Multi-Modal Medical Diagnosis via Role-Specialized Collaborationcited by 40
  23. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  24. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 39
  25. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
  26. 2025MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoningcited by 35
  27. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  28. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  29. 2024CodeMind: Evaluating Large Language Models for Code Reasoningcited by 32
  30. 2024GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generationcited by 32
  31. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  32. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
  33. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  34. 2024LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMscited by 25
  35. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoningcited by 25
  36. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 25
  37. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  38. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
  2. 2025MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task
  3. 2024DotaMath: Decomposition of Thought with Code Assistance and Self-correction for Mathematical Reasoning
  4. 2024CoinMath: Harnessing the Power of Coding Instruction for Math LLMs
  5. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data
  6. 2025Evaluation of LLMs for mathematical problem solving
  7. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time
  8. 2024Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions