mlsanityData to 14 Oct 2026

Adaptive Tool Generation with Models as Tools and Reinforcement Learning

arXiv 2510.06825 ↗First posted 8 Oct 2025cs.CLcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

also in 1 more
Abstract (arXiv)

Tool-augmented language models have demonstrated strong capabilities, but their reliance on live API access creates scalability and reliability challenges during training and deployment. We propose MTR, a simulation-first training framework for tool-augmented reasoning. Instead of relying on live APIs, MTR learns from complete ReAct traces with schema-validated, simulated observations. Our approach operates through a multi-agent architecture where a ToolMaker generates task-specific, OpenAI-compatible tool interfaces, an AutoAgent produces structured think-act-observe sequences, and a ToolActor simulates realistic responses. Training proceeds in two stages: Stage-1 Supervised Fine-Tuning (SFT) teaches 'trace grammar' from complete reasoning sequences; Stage-2 Group Relative Policy Optimization (GRPO) optimizes strategy with a composite trace reward that balances answer correctness and internal consistency. Across four multi-hop QA benchmarks (HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle), MTR attains competitive Exact Match (EM) scores to live-API systems and excels on reasoning-intensive tasks, suggesting that effective tool reasoning can be learned from structured traces without live interactions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 33 of its 39 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

33 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  3. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 29

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
17 more
  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  5. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  6. 2022PAL: Program-aided Language Modelscited by 652
  7. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  8. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  9. 2025Agentic Reinforced Policy Optimizationcited by 172
  10. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
  11. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  12. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 116
  13. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Datasetcited by 51
  14. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  15. 2025RecoWorld: Building Simulated Environments for Agentic Recommender Systemscited by 19
  16. 2025Generalizable End-to-End Tool-Use RL with Synthetic CodeGymcited by 17
  17. 2025SSRL: Self-Search Reinforcement Learningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Joint Optimization of Tool Creation and Use for Large Language Model Agents
  2. 2025Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning
  3. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
  4. 2026AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization
  5. 2025A2FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning
  6. 2026ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory
  7. 2026OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction
  8. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents