mlsanityData to 7 Oct 2026

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

arXiv 2403.04746 ↗First posted 7 Mar 2024cs.CL cs.AI cs.LGcited by 19

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 43 of its 69 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

43 of 69 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Don't Fine-Tune, Decode: Syntax Error-Free Tool Use via Constrained Decoding10 Oct
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  3. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  4. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  5. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb
  7. 2022TALM: Tool Augmented Language Models24 May

Other references · 36

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
24 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,080
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  3. 2023Teaching Large Language Models to Self-Debugcited by 963
  4. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  5. 2022Large Language Models Can Self-Improvecited by 656
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  7. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  8. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  9. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  10. 2023Language Models can Solve Computer Taskscited by 371
  11. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  12. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  13. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  14. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  15. 2021Generating Datasets with Pretrained Language Modelscited by 192
  16. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  17. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  18. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  19. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  20. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  21. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  22. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  23. 2024Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environmentscited by 39
  24. 2022The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decodingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 8.

19 papers.

Sort:

By area:

  1. 2024Tool Learning with Large Language Models: A Surveycited by 221
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  3. 2024LLM With Tools: A Surveycited by 60
  4. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  5. 2024Learning Evolving Tools for Large Language Modelscited by 15
  6. 2026SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scalecited by 15
  7. 2024Almanac Copilot: Towards Autonomous Electronic Health Record Navigationcited by 9
  8. 2025The Curious Language Model: Strategic Test-Time Information Acquisitioncited by 5
  9. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  10. 2026OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interactioncited by 3
  11. 2025Head-Specific Intervention Can Induce Misaligned AI Coordination in Large Language Modelscited by 1
  12. 2025CarbonCall: Sustainability-Aware Function Calling for Large Language Models on Edge Devicescited by 1
7 more
  1. 2025MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuningcited by 1
  2. 2026Tool Retrieval Bridge: Aligning Vague Instructions with Retriever Preferences via Bridge Modelcited by 1
  3. 2026Thinking Is Not Telling: Information Disclosure in User-Service LLM Agentscited by 0
  4. 2026GRAFT: Graph-Tokenized LLMs for Tool Planningcited by 0
  5. 2026ParaTool: Shifting Tool Representations from Context to Parameterscited by 0
  6. 2026When Does Action Credit Need Updating?cited by 0
  7. 2026Self-Evolving Agents via Likelihood-Guided Tool-Space Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction
  2. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  3. 2026Toolcompass: Guiding Tool Trialing, Not Suppressing It
  4. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents
  5. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
  6. 2024Towards Practical Tool Usage for Continually Learning LLMs
  7. 2026Try, Check and Retry: A Divide-and-Conquer Framework for Boosting Long-context Tool-Calling Performance of LLMs
  8. 2026ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs