# Environment Evolution for Terminal Agents

arXiv: https://arxiv.org/abs/2609.04128 · HTML: https://mlsanity.com/paper/2609.04128/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 27 of 27)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions](https://mlsanity.com/paper/2003.08536/) (2020)
- [AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation](https://mlsanity.com/paper/2408.00764/) (2024)
- [Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers](https://mlsanity.com/paper/2510.11370/) (2025)
- [Endless Terminals: Scaling RL Environments for Terminal Agents](https://mlsanity.com/paper/2601.16443/) (2026)
- [SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent](https://mlsanity.com/paper/2511.16108/) (2025)
- [On Data Engineering for Scaling LLM Terminal Capabilities](https://mlsanity.com/paper/2602.21193/) (2026)
- [GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators](https://mlsanity.com/paper/2512.19682/) (2025)
- [Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments](https://mlsanity.com/paper/2602.01244/) (2026)
- [CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution](https://mlsanity.com/paper/2604.15840/) (2026)
- [Toward Scalable Terminal Task Synthesis via Skill Graphs](https://mlsanity.com/paper/2604.25727/) (2026)
- [Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned](https://mlsanity.com/paper/2603.05344/) (2026)
- [What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design](https://mlsanity.com/paper/2604.28093/) (2026)
- [Learning Generalizable Behaviors for Terminal Agents](https://mlsanity.com/paper/2608.22631/) (2026)
- [COvolve: Adversarial Co-Evolution of Large-Language-Model-Generated Policies and Environments via Two-Player Zero-Sum Game](https://mlsanity.com/paper/2603.28386/) (2026)

## Built on by (most cited first, 2 of 2)

- [ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents](https://mlsanity.com/paper/2609.17523/) (2026)
- [AutoDataBench: Can Agents Write the Data That Feeds the Self-Improvement Loop?](https://mlsanity.com/paper/2609.35025/) (2026)

## Abstract (arXiv)

Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.