# Procedural Environment Generation for Tool-Use Agents

arXiv: https://arxiv.org/abs/2506.11045 · HTML: https://mlsanity.com/paper/2506.11045/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 24 of 24)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [xLAM: A Family of Large Action Models to Empower AI Agent Systems](https://mlsanity.com/paper/2409.03215/) (2024)
- [Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios](https://mlsanity.com/paper/2401.17167/) (2024)
- [AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation](https://mlsanity.com/paper/2408.00764/) (2024)
- [Large Language Models Empowered Personalized Web Agents](https://mlsanity.com/paper/2410.17236/) (2024)
- [NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls](https://mlsanity.com/paper/2409.03797/) (2024)
- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [OpenResearcher: Unleashing AI for Accelerated Scientific Research](https://mlsanity.com/paper/2408.06941/) (2024)

## Built on by (most cited first, 24 of 24)

- [Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.10090/) (2026)
- [EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis](https://mlsanity.com/paper/2601.05808/) (2026)
- [Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence](https://mlsanity.com/paper/2604.18292/) (2026)
- [EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL](https://mlsanity.com/paper/2605.18703/) (2026)
- [Environment Scaling for Interactive Agentic Experience Collection: A Survey](https://mlsanity.com/paper/2511.09586/) (2025)
- [PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems](https://mlsanity.com/paper/2606.22388/) (2026)
- [GRPO is Secretly a Process Reward Model](https://mlsanity.com/paper/2509.21154/) (2025)
- [ClawEnvKit: Automatic Environment Generation for Claw-Like Agents](https://mlsanity.com/paper/2604.18543/) (2026)
- [SynthTools: A Framework for Scaling Synthetic Tools for Agent Development](https://mlsanity.com/paper/2511.09572/) (2025)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training](https://mlsanity.com/paper/2603.01714/) (2026)
- [JTPRO: A Joint Tool-Prompt Reflective Optimization Framework for Language Agents](https://mlsanity.com/paper/2604.19821/) (2026)
- [TRACE: Capability-Targeted Agentic Training](https://mlsanity.com/paper/2604.05336/) (2026)
- [Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2604.07146/) (2026)
- [Scalable Environments Drive Generalizable Agents](https://mlsanity.com/paper/2605.18181/) (2026)
- [Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model](https://mlsanity.com/paper/2604.17739/) (2026)
- [Accelerating Constrained Decoding with Token Space Compression](https://mlsanity.com/paper/2605.29986/) (2026)
- [Failure-Guided Co-Evolution of Prompts and Training Data](https://mlsanity.com/paper/2609.15209/) (2026)
- [Toolcompass: Guiding Tool Trialing, Not Suppressing It](https://mlsanity.com/paper/2609.25678/) (2026)
- [When Does Action Credit Need Updating?](https://mlsanity.com/paper/2609.29007/) (2026)
- [WEFT: Scaling Tool-Use Post-Training for General-Purpose Agents](https://mlsanity.com/paper/2609.36887/) (2026)
- [ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.05935/) (2026)
- [Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds](https://mlsanity.com/paper/2610.14860/) (2026)
- [One Task, Many Interfaces: Simple Cross-Interface Consistency Training Improves LLM Agent Robustness](https://mlsanity.com/paper/2610.20576/) (2026)

## Abstract (arXiv)

Although the power of LLM tool-use agents has ignited a flurry of recent research in this area, the curation of tool-use training data remains an open problem-especially for online RL training. Existing approaches to synthetic tool-use data generation tend to be non-interactive, and/or non-compositional. We introduce RandomWorld, a pipeline for the procedural generation of interactive tools and compositional tool-use data. We show that models tuned via SFT and RL on synthetic RandomWorld data improve on a range of tool-use benchmarks, and set the new SoTA for two metrics on the NESTFUL dataset. Further experiments show that downstream performance scales with the amount of RandomWorld-generated training data, opening up the possibility of further improvement through the use of entirely synthetic data.