mlsanityData to 14 Oct 2026

Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments

arXiv 2508.08791 ↗First posted 12 Aug 2025cs.CL cs.AIcited by 19

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Effective tool use is essential for large language models (LLMs) to interact with their environment. However, progress is limited by the lack of efficient reinforcement learning (RL) frameworks specifically designed for tool use, due to challenges in constructing stable training environments and designing verifiable reward mechanisms. To address this, we propose an automated environment construction pipeline, incorporating scenario decomposition, document generation, function integration, complexity scaling, and localized deployment. This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools. Additionally, we introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution. When combined with trajectory data collected from the constructed environments, this mechanism integrates seamlessly with standard RL algorithms to facilitate feedback-driven model training. Experiments on LLMs of varying scales demonstrate that our approach significantly enhances the models' tool-use performance without degrading their general capabilities. Our analysis suggests that these gains result from improved context understanding and reasoning, driven by updates to the lower-layer MLP parameters in models. Code and data are available at https://github.com/bytedance/FTRL.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 43 of its 60 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

43 of 60 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025ToolRL: Reward is All Tool Learning Needs16 Apr

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2024Qwen2.5 Technical Reportcited by 4,942
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
30 more
  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  10. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
  11. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  12. 2023Tool Learning with Foundation Modelscited by 302
  13. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  14. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  15. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  16. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  17. 2024Tool Learning with Large Language Models: A Surveycited by 223
  18. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  19. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  20. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 105
  21. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  22. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 82
  23. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 74
  24. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  25. 2023ControlLLM: Augment Language Models with Tools by Searching on Graphscited by 59
  26. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 50
  27. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 47
  28. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  29. 2025MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Modelscited by 15
  30. 2024TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Usecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 13.

19 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025Towards General Agentic Intelligence via Environment Scalingcited by 54
  3. 2026ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Trainingcited by 32
  4. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systemscited by 22
  5. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  6. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RLcited by 14
  7. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  8. 2026MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matchingcited by 10
  9. 2026ASTRA: Automated Synthesis of agentic Trajectories and Reinforcement Arenascited by 8
  10. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  11. 2026CCTU: A Benchmark for Tool Use under Complex Constraintscited by 7
  12. 2025SynthTools: A Framework for Scaling Synthetic Tools for Agent Developmentcited by 5
7 more
  1. 2026Teaching LLMs to Learn Tool Trialing and Execution through Environment Interactioncited by 3
  2. 2026PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agentscited by 3
  3. 2026ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learningcited by 3
  4. 2026DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agentscited by 2
  5. 2026From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mixcited by 1
  6. 2026HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoningcited by 0
  7. 2026Toolcompass: Guiding Tool Trialing, Not Suppressing Itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  2. 2026Joint Optimization of Tool Creation and Use for Large Language Model Agents
  3. 2024Learning Evolving Tools for Large Language Models
  4. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
  5. 2025Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning
  6. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
  7. 2025ToolACE-R: Model-aware Iterative Training and Adaptive Refinement for Tool Learning
  8. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning