mlsanityData to 7 Oct 2026

To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization

arXiv 2502.00691 ↗First posted 2 Feb 2025cs.AI cs.CL cs.LGcited by 25

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Recent advances in mathematical problem-solving with language models (LMs) integrate chain-of-thought (CoT) reasoning and code execution to harness their complementary strengths. However, existing hybrid frameworks exhibit a critical limitation: they depend on externally dictated instructions or rigid code-integration templates, lacking metacognitive awareness -- the capacity to dynamically evaluate intrinsic capabilities and autonomously determine when and how to integrate tools. This rigidity motivates our study of autonomous code integration, enabling models to adapt tool-usage strategies as their reasoning abilities evolve during training. While reinforcement learning (RL) shows promise for boosting LLM reasoning at scale (e.g., DeepSeek-R1), we demonstrate its inefficiency in learning autonomous code integration due to inadequate exploration of the vast combinatorial space of CoT-code interleaving patterns. To address this challenge, we propose a novel Expectation-Maximization (EM) framework that synergizes structured exploration (E-step) with off-policy RL optimization (M-step), creating a self-reinforcing cycle between metacognitive tool-use decisions and evolving capabilities. Experiments reveal our method achieves superior results through improved exploration. Notably, our 7B model improves over 11% on MATH500 and 9.4% on AIME without o1-like CoT.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 21 of its 40 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

21 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training20 Jan

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  10. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  12. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
8 more
  1. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  2. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  3. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  4. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  5. 2020Learning Context-aware Task Reasoning for Efficient Meta-reinforcement Learningcited by 11
  6. 2024How Do Humans Write Code? Large Models Do It the Same Way Toocited by 11
  7. 2023Adversarial Constrained Bidding via Minimax Regret Optimization with Causality-Aware Reinforcement Learningcited by 7
  8. 2024An Empirical Study of Data Ability Boundary in LLMs' Math Reasoningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 8.

25 papers.

Sort:

By area:

  1. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  2. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  3. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  4. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  5. 2025Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RLcited by 34
  6. 2025Reverse-Engineered Reasoning for Open-Ended Generationcited by 32
  7. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
  8. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 17
  9. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  10. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 16
  11. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  12. 2026RelayLLM: Efficient Reasoning via Collaborative Decodingcited by 7
13 more
  1. 2026Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoningcited by 6
  2. 2026Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generationcited by 6
  3. 2025Attributes as Textual Genes: Leveraging LLMs as Genetic Algorithm Simulators for Conditional Synthetic Data Generationcited by 5
  4. 2025Teaching Language Models to Reason with Toolscited by 5
  5. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 4
  6. 2025CogDoc: Towards Unified thinking in Documentscited by 3
  7. 2025Fast, Slow, and Tool-augmented Thinking for LLMs: A Reviewcited by 2
  8. 2025Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsingcited by 1
  9. 2025UniRel: Relation-Centric Knowledge Graph Question Answering with RL-Tuned LLM Reasoningcited by 1
  10. 2026STAR-P'olyaMath: Multi-Agent Reasoning under Persistent Meta-Strategic Supervisioncited by 1
  11. 2026Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidthcited by 0
  12. 2026Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learningcited by 0
  13. 2026SpecPath: Testing Coding Agents Across Contract-Equivalent Specification Historiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Effective Code-Integrated Reasoning
  2. 2026Teaching Language Models to Think in Code
  3. 2026Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
  4. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
  5. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving
  6. 2025Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving
  7. 2025Understanding Tool-Integrated Reasoning
  8. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning