# Acting Less is Reasoning More! Teaching Model to Act Efficiently

arXiv: https://arxiv.org/abs/2504.14870 · HTML: https://mlsanity.com/paper/2504.14870/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 21 of 21)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.19470/) (2025)
- [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://mlsanity.com/paper/2410.18982/) (2024)
- [CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models](https://mlsanity.com/paper/2305.14318/) (2023)
- [SMART: Self-Aware Agent for Tool Overuse Mitigation](https://mlsanity.com/paper/2502.11435/) (2025)
- [Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees](https://mlsanity.com/paper/2406.07115/) (2024)
- [Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.24377/) (2025)
- [Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger](https://mlsanity.com/paper/2502.12961/) (2025)
- [Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model](https://mlsanity.com/paper/2310.05155/) (2023)
- [Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution](https://mlsanity.com/paper/2401.13996/) (2024)
- [AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction](https://mlsanity.com/paper/2410.19743/) (2024)
- [SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration](https://mlsanity.com/paper/2412.12151/) (2024)
- [Integrating Pretrained Language Model for Dialogue Policy Learning](https://mlsanity.com/paper/2111.01398/) (2021)

## Built on by (most cited first, 30 of 80)

- [Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning](https://mlsanity.com/paper/2508.19828/) (2025)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Agentic Reinforced Policy Optimization](https://mlsanity.com/paper/2507.19849/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment](https://mlsanity.com/paper/2505.11821/) (2025)
- [UserBench: An Interactive Gym Environment for User-Centric Agents](https://mlsanity.com/paper/2507.22034/) (2025)
- [Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning](https://mlsanity.com/paper/2505.16410/) (2025)
- [In-the-Flow Agentic System Optimization for Effective Planning and Tool Use](https://mlsanity.com/paper/2510.05592/) (2025)
- [Deep Research: A Survey of Autonomous Research Agents](https://mlsanity.com/paper/2508.12752/) (2025)
- [AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgeting](https://mlsanity.com/paper/2505.18822/) (2025)
- [From Word to World: Can Large Language Models be Implicit Text-based World Models?](https://mlsanity.com/paper/2512.18832/) (2025)
- [CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents](https://mlsanity.com/paper/2511.02734/) (2025)
- [UserRL: Training Interactive User-Centric Agent via Reinforcement Learning](https://mlsanity.com/paper/2509.19736/) (2025)
- [Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.26684/) (2026)
- [SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph](https://mlsanity.com/paper/2510.20022/) (2025)
- [Leanabell-Prover-V2: Verifier-integrated Reasoning for Formal Theorem Proving via Reinforcement Learning](https://mlsanity.com/paper/2507.08649/) (2025)
- [ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learning](https://mlsanity.com/paper/2508.09303/) (2025)
- [Time-R1: Towards Comprehensive Temporal Reasoning in LLMs](https://mlsanity.com/paper/2505.13508/) (2025)
- [Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents](https://mlsanity.com/paper/2510.23691/) (2025)
- [Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary](https://mlsanity.com/paper/2506.00886/) (2025)
- [Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learning](https://mlsanity.com/paper/2509.23285/) (2025)
- [xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning](https://mlsanity.com/paper/2510.08439/) (2025)
- [Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning](https://mlsanity.com/paper/2510.23038/) (2025)
- [From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory](https://mlsanity.com/paper/2511.07800/) (2025)
- [AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent](https://mlsanity.com/paper/2512.20745/) (2025)
- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [Toward Efficient Agents: Memory, Tool learning, and Planning](https://mlsanity.com/paper/2601.14192/) (2026)
- [Mem2Evolve: Towards Self-Evolving Agents via Co-Evolutionary Capability Expansion and Experience Distillation](https://mlsanity.com/paper/2604.10923/) (2026)

## Abstract (arXiv)

Tool-integrated reasoning (TIR) augments large language models (LLMs) with the ability to invoke external tools during long-form reasoning, such as search engines and code interpreters, to solve tasks beyond the capabilities of internal reasoning. While reinforcement learning (RL) has shown promise in training such agents, most of existing approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use. This often leads to excessive tool calling, incurring high computational costs and hindering the development of internal reasoning capabilities - a phenomenon known as textitcognitive offloading. To this end, we propose Optimal Tool Call-controlled Policy Optimization (OTC-PO), a simple yet effective RL-based framework that encourages models to produce accurate answers with minimal tool calls. Our method introduces a tool-integrated reward that jointly considers answer correctness and corresponding tool use behavior of model to reach that answer. To validate the effectiveness, we introduce the metric of textittool productivity, defined as the ratio between the number of correct answers and the total number of tool calls across all test cases. This metric reflects how efficiently tool usage contributes to successful task completion, with higher values indicating smarter and more autonomous reasoning. We instantiate this framework within both Proximal Policy Optimization (PPO) and Group Relative Preference Optimization (GRPO), resulting in OTC-PPO and OTC-GRPO. Experiments with Qwen-2.5 and Qwen-Math across multiple QA benchmarks show that our approach reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4%, while maintaining comparable answer accuracy.