mlsanityData to 14 Oct 2026

Replacing thinking with tool usage enables reasoning in small language models

arXiv 2507.05065 ↗First posted 7 Jul 2025cs.LG cs.AIcited by 1

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Recent advances have established a new machine learning paradigm based on scaling up compute at inference time as well as at training time. In that line of work, a combination of Supervised Fine-Tuning (SFT) on synthetic demonstrations and Reinforcement Learning with Verifiable Rewards (RLVR) is used for training Large Language Models to expend extra compute during inference in the form of "thoughts" expressed in natural language. In this paper, we propose to instead format these tokens as a multi-turn interaction trace with a stateful tool. At each turn, the new state of the tool is appended to the context of the model, whose job is to generate the tokens necessary to control the tool via a custom DSL. We benchmark this approach on the problem of repairing malfunctioning Python code, and show that this constrained setup allows for faster sampling of experience and a denser reward signal, allowing even models of size up to 3B parameters to learn how to proficiently expend additional compute on the task.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 39 of its 48 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

39 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning2 Oct
  3. 2024NExT: Teaching Large Language Models to Reason about Code Execution23 Apr
  4. 2023Teaching Large Language Models to Self-Debug11 Apr
  5. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 34

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
22 more
  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  2. 2025LIMO: Less is More for Reasoningcited by 498
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  4. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  5. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  6. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  7. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  8. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  9. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  10. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 159
  11. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 136
  12. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  13. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 74
  14. 2024TinyAgent: Function Calling at the Edgecited by 58
  15. 2025START: Self-taught Reasoner with Toolscited by 47
  16. 2020Experimental design for MRI by greedy policy searchcited by 40
  17. 2020Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesiscited by 39
  18. 2020Estimating Gradients for Discrete Random Variables by Sampling without Replacementcited by 31
  19. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  20. 2024Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoningcited by 26
  21. 2024CodeEditorBench: Evaluating Code Editing Capability of Large Language Modelscited by 23
  22. 2024Training Language Models on Synthetic Edit Sequences Improves Code Synthesiscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21

Similar, not cited

Close in wording, no citation either way

  1. 2025Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement
  2. 2025Understanding Tool-Integrated Reasoning
  3. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  4. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  5. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
  6. 2023ART: Automatic multi-step reasoning and tool-use for large language models
  7. 2025The Challenge of Teaching Reasoning to LLMs Without RL or Distillation
  8. 2025Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models