# To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization

arXiv: https://arxiv.org/abs/2502.00691 · HTML: https://mlsanity.com/paper/2502.00691/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 2 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [Thinking Fast and Slow with Deep Learning and Tree Search](https://mlsanity.com/paper/1705.08439/) (2017)
- [OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset](https://mlsanity.com/paper/2402.10176/) (2024)
- [Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training](https://mlsanity.com/paper/2501.11425/) (2025)
- [MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipeline](https://mlsanity.com/paper/2401.08190/) (2024)
- [Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions](https://mlsanity.com/paper/2205.14318/) (2022)
- [Learning Context-aware Task Reasoning for Efficient Meta-reinforcement Learning](https://mlsanity.com/paper/2003.01373/) (2020)
- [How Do Humans Write Code? Large Models Do It the Same Way Too](https://mlsanity.com/paper/2402.15729/) (2024)
- [Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learning](https://mlsanity.com/paper/2306.07106/) (2023)
- [An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning](https://mlsanity.com/paper/2403.00799/) (2024)

## Built on by (most cited first, 25 of 25)

- [VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use](https://mlsanity.com/paper/2509.01055/) (2025)
- [Parallel-R1: Towards Parallel Thinking via Reinforcement Learning](https://mlsanity.com/paper/2509.07980/) (2025)
- [Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning](https://mlsanity.com/paper/2509.03646/) (2025)
- [UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learning](https://mlsanity.com/paper/2510.13515/) (2025)
- [Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL](https://mlsanity.com/paper/2505.17952/) (2025)
- [Reverse-Engineered Reasoning for Open-Ended Generation](https://mlsanity.com/paper/2509.06160/) (2025)
- [MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Data](https://mlsanity.com/paper/2603.09206/) (2026)
- [HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocation](https://mlsanity.com/paper/2505.19866/) (2025)
- [Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing](https://mlsanity.com/paper/2510.15349/) (2025)
- [AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent](https://mlsanity.com/paper/2512.20745/) (2025)
- [From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning](https://mlsanity.com/paper/2511.23031/) (2025)
- [RelayLLM: Efficient Reasoning via Collaborative Decoding](https://mlsanity.com/paper/2601.05167/) (2026)
- [Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation](https://mlsanity.com/paper/2607.05382/) (2026)
- [Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning](https://mlsanity.com/paper/2605.14054/) (2026)
- [StatEval: A Comprehensive Benchmark for Large Language Models in Statistics](https://mlsanity.com/paper/2510.09517/) (2025)
- [Attributes as Textual Genes: Leveraging LLMs as Genetic Algorithm Simulators for Conditional Synthetic Data Generation](https://mlsanity.com/paper/2509.02040/) (2025)
- [Teaching Language Models to Reason with Tools](https://mlsanity.com/paper/2510.20342/) (2025)
- [Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing](https://mlsanity.com/paper/2506.03197/) (2025)
- [CogDoc: Towards Unified thinking in Documents](https://mlsanity.com/paper/2512.12658/) (2025)
- [Fast, Slow, and Tool-augmented Thinking for LLMs: A Review](https://mlsanity.com/paper/2508.12265/) (2025)
- [UniRel: Relation-Centric Knowledge Graph Question Answering with RL-Tuned LLM Reasoning](https://mlsanity.com/paper/2512.17043/) (2025)
- [STAR-P'olyaMath: Multi-Agent Reasoning under Persistent Meta-Strategic Supervision](https://mlsanity.com/paper/2605.19338/) (2026)
- [SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Histories](https://mlsanity.com/paper/2608.09799/) (2026)
- [Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth](https://mlsanity.com/paper/2605.18603/) (2026)
- [Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning](https://mlsanity.com/paper/2606.06835/) (2026)

## Abstract (arXiv)

Recent advances in mathematical problem-solving with language models (LMs) integrate chain-of-thought (CoT) reasoning and code execution to harness their complementary strengths. However, existing hybrid frameworks exhibit a critical limitation: they depend on externally dictated instructions or rigid code-integration templates, lacking metacognitive awareness -- the capacity to dynamically evaluate intrinsic capabilities and autonomously determine when and how to integrate tools. This rigidity motivates our study of autonomous code integration, enabling models to adapt tool-usage strategies as their reasoning abilities evolve during training. While reinforcement learning (RL) shows promise for boosting LLM reasoning at scale (e.g., DeepSeek-R1), we demonstrate its inefficiency in learning autonomous code integration due to inadequate exploration of the vast combinatorial space of CoT-code interleaving patterns. To address this challenge, we propose a novel Expectation-Maximization (EM) framework that synergizes structured exploration (E-step) with off-policy RL optimization (M-step), creating a self-reinforcing cycle between metacognitive tool-use decisions and evolving capabilities. Experiments reveal our method achieves superior results through improved exploration. Notably, our 7B model improves over 11% on MATH500 and 9.4% on AIME without o1-like CoT.