mlsanityData to 14 Oct 2026

AutoForge: Automated Environment Synthesis for Agentic Reinforcement Learning

arXiv 2512.22857 ↗First posted 28 Dec 2025cs.CL cs.AIcited by 17

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Conducting reinforcement learning (RL) in simulated environments offers a cost-effective and highly scalable way to enhance language-based agents. However, previous work has been limited to semi-automated environment synthesis or tasks lacking sufficient difficulty, offering little breadth or depth. In addition, the instability of simulated users integrated into these environments, along with the heterogeneity across simulated environments, poses further challenges for agentic RL. In this work, we propose: (1) a unified pipeline for automated and scalable synthesis of simulated environments associated with high-difficulty but easily verifiable tasks; and (2) an environment level RL algorithm that not only effectively mitigates user instability but also performs advantage estimation at the environment level, thereby improving training efficiency and stability. Comprehensive evaluations on agentic benchmarks, including tau-bench, tau2-Bench, and VitaBench, validate the effectiveness of our proposed method. Further in-depth analyses underscore its out-of-domain generalization.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 15 of its 27 references that are in the map. First posted on arXiv 28 Dec 2025.

Builds on

15 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025GTM: Simulating the World of Tools for AI Agents4 Dec
  2. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning10 Sep
  3. 2025ToolRL: Reward is All Tool Learning Needs16 Apr

Other references · 12

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  5. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  6. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  7. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  8. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  9. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
  10. 2025Scaling Agents via Continual Pre-trainingcited by 40
  11. 2025Generalizable End-to-End Tool-Use RL with Synthetic CodeGymcited by 17
  12. 2025ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 63
  2. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 28
  3. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 24
  4. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RLcited by 14
  5. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 12
  6. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 8
  7. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  8. 2026TopoCurate:Modeling Interaction Topology for Tool-Use Agent Trainingcited by 4
  9. 2026Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learningcited by 3
  10. 2026Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Modelcited by 1
  11. 2026Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learningcited by 0
  12. 2026AgentOmnia: Scaling Agentic Models for Full-Scenario Applicationscited by 0
5 more
  1. 2026Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanismscited by 0
  2. 2026CompoWorld: Compositional Environment Scaling for General Agentscited by 0
  3. 2026ClawLoop: Self-Improving Agents via Harness-Native Training Loopcited by 0
  4. 2026Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewardscited by 0
  5. 2026Interlock: From Local Correctness to Task Completion in Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learning
  2. 2026EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agent
  3. 2025Agent2: An Agent-Generates-Agent Framework for Reinforcement Learning Automation
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  5. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
  6. 2026SkillForge: Evolving Verifiable Skills for Reinforcement Learning Agents
  7. 2026EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning
  8. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework