mlsanityData to 14 Oct 2026

Simulating Environments with Reasoning Models for Agent Training

arXiv 2511.01824 ↗First posted 3 Nov 2025cs.AI cs.LGcited by 57

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

LLM agents excel in compact environments requiring deep reasoning but remain brittle when operating in broader, more complex contexts that demand robustness across diverse tools and schemas. Building bespoke environments for training is heavy, brittle, and limits progress. In this paper, we demonstrate that LLMs can simulate realistic environment feedback without access to actual testbed data or APIs. Inspired by this capability, we propose two frameworks: Simia-SFT, a pipeline that synthesizes SFT data by amplifying small seed sets into diverse trajectories in an environment-agnostic manner, and Simia-RL, a framework that enables RL training without real environment implementations through LLM-simulated feedback. Fine-tuning open models yields consistent improvements across multiple benchmarks, surpassing GPT-4o and approaching o4-mini on τ2-Bench. Together, Simia-SFT and Simia-RL enable scalable agent training without environment engineering, replacing heavy and brittle implementations with flexible LLM-based simulation.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 24 of its 33 references that are in the map. First posted on arXiv 3 Nov 2025.

Builds on

24 of 33 references.

References · 24

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2022PAL: Program-aided Language Modelscited by 652
  11. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  12. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
12 more
  1. 2025Kimi K2: Open Agentic Intelligencecited by 567
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  4. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  5. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  6. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  7. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  8. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 80
  9. 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 70
  10. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
  11. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 42
  12. 2024ToolBridge: An Open-Source Dataset to Equip LLMs with External Tool Capabilitiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 54.

57 papers.

Sort:

By area:

  1. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 66
  2. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 63
  3. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  4. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?cited by 42
  5. 2026ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Trainingcited by 32
  6. 2026TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agentscited by 29
  7. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 28
  8. 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 26
  9. 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agentscited by 21
  10. 2025AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learningcited by 20
  11. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 18
  12. 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Usecited by 14
38 more
  1. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 14
  2. 2026Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Textcited by 13
  3. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Usecited by 13
  4. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
  5. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  6. 2026OpenComputer: Verifiable Software Worlds for Computer-Use Agentscited by 10
  7. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 8
  8. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  9. 2026OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulationcited by 7
  10. 2026EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repaircited by 6
  11. 2025SynthTools: A Framework for Scaling Synthetic Tools for Agent Developmentcited by 5
  12. 2026MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correctioncited by 5
  13. 2026Beyond Quantity: Trajectory Diversity Scaling for Code Agentscited by 5
  14. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 5
  15. 2026TopoCurate:Modeling Interaction Topology for Tool-Use Agent Trainingcited by 4
  16. 2026CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verificationcited by 4
  17. 2026ARM: Role-Conditioned Neuron Transplantation for Training-Free Generalist LLM Agent Mergingcited by 3
  18. 2026ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learningcited by 3
  19. 2026What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agentscited by 3
  20. 2026MagicAgent: Towards Generalized Agent Planningcited by 2
  21. 2026Scalable Environments Drive Generalizable Agentscited by 2
  22. 2026WRIT: Write-Read Intensive Trajectory Synthesis for Multi-Turn User-Facing Agentscited by 2
  23. 2026RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agentscited by 2
  24. 2026DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environmentcited by 2
  25. 2026Simulate to Generalize: Scaling Stateful Supervision for API-calling Agents using LLM World Modelscited by 2
  26. 2026EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learningcited by 2
  27. 2026AgentBrew: Offline Tool-Use Agent Learning from Raw Real-World Trajectoriescited by 2
  28. 2026Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Modelcited by 1
  29. 2026EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulationcited by 1
  30. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1
  31. 2026Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Contextcited by 1
  32. 2026MidTool: Mid-training Data Synthesis for Agentic Tool Usecited by 1
  33. 2026EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agentcited by 1
  34. 2026Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agentscited by 1
  35. 2026Implicit Intelligence -- Evaluating Agents on What Users Don't Saycited by 0
  36. 2026Agent Seer: Synthesizing Scenarios from Specification Understandingcited by 0
  37. 2026WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learningcited by 0
  38. 2026Agent-Editing World Model: Rethinking World Modeling for LLM Agentscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Simple Test-Time Environments for LLM Web Agents
  2. 2025Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback
  3. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
  4. 2026State2State: Environment-Derived Mid-Training for LLM Agents
  5. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning
  6. 2026PhantomEnvironments: Training LLM Agents in Fictional Worlds
  7. 2025Test-Time Adaptation for LLM Agents via Environment Interaction
  8. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL