# EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection

arXiv: https://arxiv.org/abs/2603.04900 · HTML: https://mlsanity.com/paper/2603.04900/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 5 Mar 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)
- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 17 of 17)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction](https://mlsanity.com/paper/2401.06201/) (2024)
- [AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls](https://mlsanity.com/paper/2402.04253/) (2024)
- [A Systematic Survey of Automatic Prompt Optimization Techniques](https://mlsanity.com/paper/2502.16923/) (2025)
- [RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents](https://mlsanity.com/paper/2507.22844/) (2025)
- [From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions](https://mlsanity.com/paper/2410.08197/) (2024)

## Built on by (most cited first, 24 of 24)

- [SEAL: Synergistic Co-Evolution of Agents and Learning Environments](https://mlsanity.com/paper/2605.24426/) (2026)
- [StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2604.09000/) (2026)
- [Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimization](https://mlsanity.com/paper/2604.07165/) (2026)
- [Recursive Self-Evolving Agents via Held-Out Selection](https://mlsanity.com/paper/2606.28374/) (2026)
- [When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models](https://mlsanity.com/paper/2603.23085/) (2026)
- [AEL: Evolving Agent Harness in Open-Ended Environments](https://mlsanity.com/paper/2604.21725/) (2026)
- [ActorMind: Emulating Human Actor Reasoning for Speech Role-Playing](https://mlsanity.com/paper/2604.11103/) (2026)
- [Living-Harness Is an Interactive-Agent Evolver](https://mlsanity.com/paper/2607.26598/) (2026)
- [AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?](https://mlsanity.com/paper/2608.00155/) (2026)
- [SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation](https://mlsanity.com/paper/2603.16161/) (2026)
- [Environment-Grounded Automated Prompt Optimization for LLM Game Agents](https://mlsanity.com/paper/2606.17838/) (2026)
- [EvoUndo: Recoverability-Constrained Self-Evolution for LLM Agent Harnesses](https://mlsanity.com/paper/2608.28363/) (2026)
- [From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs](https://mlsanity.com/paper/2605.02130/) (2026)
- [Coverage, Not Credit: Failure-Credit Routing of Zeroth-Order Perturbation Budgets Does Not Improve On-Pool Sample Efficiency for LLM Agents](https://mlsanity.com/paper/2608.28011/) (2026)
- [CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents](https://mlsanity.com/paper/2608.30147/) (2026)
- [Agents in the Large: Perception-Centered Architecture for Persistent Agents](https://mlsanity.com/paper/2608.30478/) (2026)
- [Where Does Harness-Optimization Value Live? Localized Gains and the Budget-Splitting Trap in Self-Evolving LLM Agents](https://mlsanity.com/paper/2609.02889/) (2026)
- [Learnable Instance Attention Filtering for Adaptive Detector Distillation](https://mlsanity.com/paper/2603.26088/) (2026)
- [Agent-GWO: Collaborative Agents for Dynamic Prompt Optimization in Large Language Models](https://mlsanity.com/paper/2604.18612/) (2026)
- [From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL](https://mlsanity.com/paper/2608.13787/) (2026)
- [Bounded Personas Match Retrieval on Classification but Not Regression for a Frozen Agent](https://mlsanity.com/paper/2609.02890/) (2026)
- [At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent](https://mlsanity.com/paper/2609.04217/) (2026)
- [Self-Evolving Agents via Likelihood-Guided Tool-Space Optimization](https://mlsanity.com/paper/2609.34151/) (2026)
- [Coupled Harness--Model Evolution for Scientific Equation Discovery](https://mlsanity.com/paper/2610.17740/) (2026)

## Abstract (arXiv)

LLM-based agents depend on effective tool-use policies to solve complex tasks, yet optimizing these policies remains challenging due to delayed supervision and the difficulty of credit assignment in long-horizon trajectories. Existing optimization approaches tend to be either monolithic, which are prone to entangling behaviors, or single-aspect, which ignore cross-module error propagation. To address these limitations, we propose EvoTool, a self-evolving framework that optimizes a modular tool-use policy via a gradient-free evolutionary paradigm. EvoTool decomposes agent's tool-use policy into four modules, including Planner, Selector, Caller, and Synthesizer, and iteratively improves them in a self-improving loop through three novel mechanisms. Trajectory-Grounded Blame Attribution uses diagnostic traces to localize failures to a specific module. Feedback-Guided Targeted Mutation then edits only that module via natural-language critique. Diversity-Aware Population Selection preserves complementary candidates to ensure solution diversity. Across four benchmarks, EvoTool outperforms strong baselines by over 5 points on both GPT-4.1 and Qwen3-8B, while achieving superior efficiency and transferability. The code will be released once paper is accepted.