mlsanityData to 7 Oct 2026

Environment Evolution for Terminal Agents

arXiv 2609.04128 ↗First posted 3 Sep 2026cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 37 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

27 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Learning Generalizable Behaviors for Terminal Agents23 Aug
  2. 2026CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution17 Apr
  3. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent20 Nov

Other references · 24

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  4. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  5. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  6. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  7. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  8. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  9. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  10. 2020Prioritized Level Replaycited by 190
  11. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  12. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
12 more
  1. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  2. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  3. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  4. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  5. 2026Endless Terminals: Scaling RL Environments for Terminal Agentscited by 43
  6. 2025GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulatorscited by 34
  7. 2026On Data Engineering for Scaling LLM Terminal Capabilitiescited by 34
  8. 2026Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environmentscited by 27
  9. 2026Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learnedcited by 18
  10. 2026Toward Scalable Terminal Task Synthesis via Skill Graphscited by 16
  11. 2026What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Designcited by 3
  12. 2026COvolve: Adversarial Co-Evolution of Large-Language-Model-Generated Policies and Environments via Two-Player Zero-Sum Gamecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agentscited by 3
  2. 2026AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SEAL: Synergistic Co-Evolution of Agents and Learning Environments
  2. 2026VERA: Scaling Verifiable Environments for Agentic co-Evolution
  3. 2026ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learning
  4. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence
  5. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application
  6. 2026Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
  7. 2026State2State: Environment-Derived Mid-Training for LLM Agents
  8. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation