# LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

arXiv: https://arxiv.org/abs/2403.04746 · HTML: https://mlsanity.com/paper/2403.04746/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 43 of its 69 references that are in the map.
- First posted on arXiv 7 Mar 2024.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 30 of 43)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [ExpeL: LLM Agents Are Experiential Learners](https://mlsanity.com/paper/2308.10144/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Effective Long-Context Scaling of Foundation Models](https://mlsanity.com/paper/2309.16039/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023)

## Built on by (most cited first, 19 of 19)

- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [LLM With Tools: A Survey](https://mlsanity.com/paper/2409.18807/) (2024)
- [From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions](https://mlsanity.com/paper/2410.08197/) (2024)
- [Learning Evolving Tools for Large Language Models](https://mlsanity.com/paper/2410.06617/) (2024)
- [SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scale](https://mlsanity.com/paper/2606.03056/) (2026)
- [Almanac Copilot: Towards Autonomous Electronic Health Record Navigation](https://mlsanity.com/paper/2405.07896/) (2024)
- [The Curious Language Model: Strategic Test-Time Information Acquisition](https://mlsanity.com/paper/2506.09173/) (2025)
- [Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism](https://mlsanity.com/paper/2510.13170/) (2025)
- [OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction](https://mlsanity.com/paper/2602.15197/) (2026)
- [Tool Retrieval Bridge: Aligning Vague Instructions with Retriever Preferences via Bridge Model](https://mlsanity.com/paper/2604.07816/) (2026)
- [Head-Specific Intervention Can Induce Misaligned AI Coordination in Large Language Models](https://mlsanity.com/paper/2502.05945/) (2025)
- [CarbonCall: Sustainability-Aware Function Calling for Large Language Models on Edge Devices](https://mlsanity.com/paper/2504.20348/) (2025)
- [MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning](https://mlsanity.com/paper/2507.21924/) (2025)
- [Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents](https://mlsanity.com/paper/2602.07796/) (2026)
- [GRAFT: Graph-Tokenized LLMs for Tool Planning](https://mlsanity.com/paper/2605.11706/) (2026)
- [ParaTool: Shifting Tool Representations from Context to Parameters](https://mlsanity.com/paper/2605.29561/) (2026)
- [When Does Action Credit Need Updating?](https://mlsanity.com/paper/2609.29007/) (2026)
- [Self-Evolving Agents via Likelihood-Guided Tool-Space Optimization](https://mlsanity.com/paper/2609.34151/) (2026)

## Abstract (arXiv)

Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.