mlsanityData to 14 Oct 2026

R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning

arXiv 2505.21668 ↗First posted 27 May 2025cs.AI cs.CL cs.SCcited by 15

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Practical guidance on training Large Language Models (LLMs) to leverage Code Interpreter across diverse tasks remains lacking. We present R1-Code-Interpreter, an extension of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and reinforcement learning (RL) to autonomously generate multiple code queries during step-by-step reasoning. Unlike prior RL + tool-use efforts focused on narrow domains such as math or retrieval, we curate 144 diverse reasoning and planning tasks and show that training a general-purpose Code Interpreter across them presents significant challenges due to task heterogeneity and scarcity of effective samples. To address this, we introduce a multi-stage curriculum learning approach that partitions training samples by measured improvement potential. The RL training prioritizes samples with higher potential and gradually shifts to lower-potential ones, increasing the average RL gains from merely +3.4% to +9.3% across Qwen-2.5 models (3/7/14B). Our final model, R1-CI-14B, improves average accuracy on the 37 test tasks from 44.1% to 72.4%, outperforming text-only GPT-4o (58.6%) and GPT-4o with Code Interpreter (70.9%). Notably, R1-CI-14B also exhibits emergent self-checking behavior through code generation. Datasets, Codes, and Models are available at https://github.com/yongchao98/R1-Code-Interpreter and https://huggingface.co/yongchao98.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 36 of its 46 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

36 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  2. 2025CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance4 Feb
  3. 2024OpenAI o1 System Card21 Dec
  4. 2024Steering Large Language Models between Code Execution and Textual Reasoning4 Oct
  5. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  6. 2024Learning to Reason via Program Generation, Emulation, and Search25 May
  7. 2023Chain of Code: Reasoning with a Language Model-Augmented Code Emulator7 Dec

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
17 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  4. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  5. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  6. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
  7. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  8. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  9. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  10. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 241
  11. 2025ToRL: Scaling Tool-Integrated RLcited by 169
  12. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  13. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  14. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  15. 2024Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Toolscited by 47
  16. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  17. 2024Can LLMs Reason in the Wild with Programs?cited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  4. 2025StepFun-Formalizer: Unlocking the Autoformalization Potential of LLMs through Knowledge-Reasoning Fusioncited by 16
  5. 2025Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoningcited by 15
  6. 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Usecited by 7
  7. 2025Learning to Reason for Hallucination Span Detectioncited by 5
  8. 2025CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questionscited by 4
  9. 2026ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibrationcited by 4
  10. 2026Rewarding Creativity: A Human-Aligned Generative Reward Model for Reinforcement Learning in Storytellingcited by 2
  11. 2026AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedbackcited by 1
  12. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
3 more
  1. 2026REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignmentcited by 1
  2. 2025Structured In-context Environment Scaling for Large Language Model Reasoningcited by 0
  3. 2026AMTFV: Agentic Mathematical Tool-Flow Verification for LLM Self-Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  2. 2025CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning
  3. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
  4. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  5. 2026Exploring Extrinsic and Intrinsic Properties for Effective Reasoning with Code Interpreter
  6. 2026Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learning
  7. 2025Code-enabled language models can outperform reasoning models on diverse tasks
  8. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning