# Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents

arXiv: https://arxiv.org/abs/2602.02050 · HTML: https://mlsanity.com/paper/2602.02050/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 29 of 29)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [Agentic Reinforced Policy Optimization](https://mlsanity.com/paper/2507.19849/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization](https://mlsanity.com/paper/2505.15107/) (2025)
- [Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning](https://mlsanity.com/paper/2505.16410/) (2025)
- [Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models](https://mlsanity.com/paper/2506.06395/) (2025)
- [RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents](https://mlsanity.com/paper/2507.22844/) (2025)
- [ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models](https://mlsanity.com/paper/2305.14323/) (2023)
- [Agentic Entropy-Balanced Policy Optimization](https://mlsanity.com/paper/2510.14545/) (2025)
- [Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens](https://mlsanity.com/paper/2505.18237/) (2025)
- [Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning](https://mlsanity.com/paper/2510.08146/) (2025)
- [Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary](https://mlsanity.com/paper/2506.00886/) (2025)
- [Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use](https://mlsanity.com/paper/2509.12867/) (2025)
- [Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning](https://mlsanity.com/paper/2509.02401/) (2025)
- [Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them](https://mlsanity.com/paper/2510.06534/) (2025)
- [WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection](https://mlsanity.com/paper/2510.18798/) (2025)
- [Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs](https://mlsanity.com/paper/2510.00861/) (2025)

## Built on by (most cited first, 5 of 5)

- [Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction](https://mlsanity.com/paper/2608.11772/) (2026)
- [Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context](https://mlsanity.com/paper/2608.17499/) (2026)
- [Semantic Uncertainty-Guided Orchestration in Hierarchical Multi-Agent Systems](https://mlsanity.com/paper/2608.14707/) (2026)
- [When Does Action Credit Need Updating?](https://mlsanity.com/paper/2609.29007/) (2026)
- [ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradients](https://mlsanity.com/paper/2609.34985/) (2026)

## Abstract (arXiv)

Tool-using agents based on Large Language Models (LLMs) excel in tasks such as mathematical reasoning and multi-hop question answering. However, in long trajectories, agents often trigger excessive and low-quality tool calls, increasing latency and degrading inference performance, making managing tool-use behavior challenging. In this work, we conduct entropy-based pilot experiments and observe a strong positive correlation between entropy reduction and high-quality tool calls. Building on this finding, we propose using entropy reduction as a supervisory signal and design two reward strategies to address the differing needs of optimizing tool-use behavior. Sparse outcome rewards provide coarse, trajectory-level guidance to improve efficiency, while dense process rewards offer fine-grained supervision to enhance performance. Experiments across diverse domains show that both reward designs improve tool-use behavior: the former reduces tool calls by 72.07% compared to the average of baselines, while the latter improves performance by 22.27%. These results position entropy reduction as a key mechanism for enhancing tool-use behavior, enabling agents to be more adaptive in real-world applications.