# Simulating Environments with Reasoning Models for Agent Training

arXiv: https://arxiv.org/abs/2511.01824 · HTML: https://mlsanity.com/paper/2511.01824/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 24 of 24)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [AgentTuning: Enabling Generalized Agent Abilities for LLMs](https://mlsanity.com/paper/2310.12823/) (2023)
- [ToolACE: Winning the Points of LLM Function Calling](https://mlsanity.com/paper/2409.00920/) (2024)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation](https://mlsanity.com/paper/2407.19056/) (2024)
- [TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments](https://mlsanity.com/paper/2510.01179/) (2025)
- [Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning](https://mlsanity.com/paper/2410.12952/) (2024)
- [ToolBridge: An Open-Source Dataset to Equip LLMs with External Tool Capabilities](https://mlsanity.com/paper/2410.10872/) (2024)

## Built on by (most cited first, 30 of 57)

- [Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.10090/) (2026)
- [EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis](https://mlsanity.com/paper/2601.05808/) (2026)
- [Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence](https://mlsanity.com/paper/2604.18292/) (2026)
- [From Word to World: Can Large Language Models be Implicit Text-based World Models?](https://mlsanity.com/paper/2512.18832/) (2025)
- [ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training](https://mlsanity.com/paper/2602.06820/) (2026)
- [TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agents](https://mlsanity.com/paper/2602.07274/) (2026)
- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [WebWorld: A Large-Scale World Model for Web Agent Training](https://mlsanity.com/paper/2602.14721/) (2026)
- [From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents](https://mlsanity.com/paper/2601.22607/) (2026)
- [AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning](https://mlsanity.com/paper/2511.19304/) (2025)
- [Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?](https://mlsanity.com/paper/2607.09996/) (2026)
- [CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Use](https://mlsanity.com/paper/2602.12268/) (2026)
- [Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond](https://mlsanity.com/paper/2604.22748/) (2026)
- [Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Text](https://mlsanity.com/paper/2601.10355/) (2026)
- [DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use](https://mlsanity.com/paper/2603.11076/) (2026)
- [InfoPO: Information-Driven Policy Optimization for User-Centric Agents](https://mlsanity.com/paper/2603.00656/) (2026)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [OpenComputer: Verifiable Software Worlds for Computer-Use Agents](https://mlsanity.com/paper/2605.19769/) (2026)
- [AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2605.29801/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation](https://mlsanity.com/paper/2604.10866/) (2026)
- [EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair](https://mlsanity.com/paper/2603.05553/) (2026)
- [SynthTools: A Framework for Scaling Synthetic Tools for Agent Development](https://mlsanity.com/paper/2511.09572/) (2025)
- [MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correction](https://mlsanity.com/paper/2601.12822/) (2026)
- [Beyond Quantity: Trajectory Diversity Scaling for Code Agents](https://mlsanity.com/paper/2602.03219/) (2026)
- [Bridging the Agent-World Gap: Text World Models for LLM-based Agents](https://mlsanity.com/paper/2606.09032/) (2026)
- [TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training](https://mlsanity.com/paper/2603.01714/) (2026)
- [CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification](https://mlsanity.com/paper/2603.01940/) (2026)
- [ARM: Role-Conditioned Neuron Transplantation for Training-Free Generalist LLM Agent Merging](https://mlsanity.com/paper/2601.07309/) (2026)
- [ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.15660/) (2026)

## Abstract (arXiv)

LLM agents excel in compact environments requiring deep reasoning but remain brittle when operating in broader, more complex contexts that demand robustness across diverse tools and schemas. Building bespoke environments for training is heavy, brittle, and limits progress. In this paper, we demonstrate that LLMs can simulate realistic environment feedback without access to actual testbed data or APIs. Inspired by this capability, we propose two frameworks: Simia-SFT, a pipeline that synthesizes SFT data by amplifying small seed sets into diverse trajectories in an environment-agnostic manner, and Simia-RL, a framework that enables RL training without real environment implementations through LLM-simulated feedback. Fine-tuning open models yields consistent improvements across multiple benchmarks, surpassing GPT-4o and approaching o4-mini on τ2-Bench. Together, Simia-SFT and Simia-RL enable scalable agent training without environment engineering, replacing heavy and brittle implementations with flexible LLM-based simulation.