# Endless Terminals: Scaling RL Environments for Terminal Agents

arXiv: https://arxiv.org/abs/2601.16443 · HTML: https://mlsanity.com/paper/2601.16443/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 24 of its 29 references that are in the map.
- First posted on arXiv 23 Jan 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- [NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System](https://mlsanity.com/paper/1802.08979/) (2018)
- [CWM: An Open-Weights LLM for Research on Code Generation with World Models](https://mlsanity.com/paper/2510.02387/) (2025)
- [Learning Formal Mathematics From Intrinsic Motivation](https://mlsanity.com/paper/2407.00695/) (2024)
- [Creating General User Models from Computer Use](https://mlsanity.com/paper/2505.10831/) (2025)
- [Scaling Agent Learning via Experience Synthesis](https://mlsanity.com/paper/2511.03773/) (2025)
- [Wait, Wait, Wait... Why Do Reasoning Models Loop?](https://mlsanity.com/paper/2512.12895/) (2025)
- [RL Grokking Recipe: How Does RL Unlock and Transfer New Algorithms in LLMs?](https://mlsanity.com/paper/2509.21016/) (2025)
- [Learning to Simulate Human Dialogue](https://mlsanity.com/paper/2601.04436/) (2026)
- [AI & Human Co-Improvement for Safer Co-Superintelligence](https://mlsanity.com/paper/2512.05356/) (2025)

## Built on by (most cited first, 30 of 54)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [Tmax: A simple recipe for terminal agents](https://mlsanity.com/paper/2606.23321/) (2026)
- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agents](https://mlsanity.com/paper/2605.25624/) (2026)
- [Toward Scalable Terminal Task Synthesis via Skill Graphs](https://mlsanity.com/paper/2604.25727/) (2026)
- [Terminal-World: Scaling Terminal-Agent Environments via Agent Skills](https://mlsanity.com/paper/2605.20876/) (2026)
- [SWE-Universe: Scale Real-World Verifiable Environments to Millions](https://mlsanity.com/paper/2602.02361/) (2026)
- [ECHO: Terminal Agents Learn World Models for Free](https://mlsanity.com/paper/2605.24517/) (2026)
- [Recursive Synthesis for Long-Horizon Terminal Tasks](https://mlsanity.com/paper/2608.05466/) (2026)
- [Failure as a Process: An Anatomy of CLI Coding Agent Trajectories](https://mlsanity.com/paper/2607.09510/) (2026)
- [Shepherd: Enabling Programmable Meta-Agents via Reversible Agentic Execution Traces](https://mlsanity.com/paper/2605.10913/) (2026)
- [CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents](https://mlsanity.com/paper/2606.22883/) (2026)
- [SETA: Scaling Environments for Terminal Agents](https://mlsanity.com/paper/2607.10891/) (2026)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks](https://mlsanity.com/paper/2605.22535/) (2026)
- [FACET: Preserving Source Intent and Executable State in Terminal Task Synthesis](https://mlsanity.com/paper/2608.18580/) (2026)
- [ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement](https://mlsanity.com/paper/2609.14857/) (2026)
- [ClawEnvKit: Automatic Environment Generation for Claw-Like Agents](https://mlsanity.com/paper/2604.18543/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks](https://mlsanity.com/paper/2608.06352/) (2026)
- [Terminal-Universe: Turning Agent Trajectories into Scalable Terminal Environments](https://mlsanity.com/paper/2609.04148/) (2026)
- [T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks](https://mlsanity.com/paper/2609.11042/) (2026)
- [Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training](https://mlsanity.com/paper/2607.27929/) (2026)
- [Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL](https://mlsanity.com/paper/2608.14312/) (2026)
- [GUI Agents with Reinforcement Learning: Toward Digital Inhabitants](https://mlsanity.com/paper/2604.27955/) (2026)
- [FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale](https://mlsanity.com/paper/2605.14445/) (2026)
- [Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL](https://mlsanity.com/paper/2606.14211/) (2026)
- [Learning Generalizable Behaviors for Terminal Agents](https://mlsanity.com/paper/2608.22631/) (2026)
- [Environment Evolution for Terminal Agents](https://mlsanity.com/paper/2609.04128/) (2026)
- [Terminus-4B: Can a Smaller Model Replace Frontier LLMs at Agentic Execution Tasks?](https://mlsanity.com/paper/2605.03195/) (2026)

## Abstract (arXiv)

Environments are the bottleneck for self-improving agents. Current terminal benchmarks were built for evaluation, not training; reinforcement learning requires a scalable pipeline, not just a dataset. We introduce Endless Terminals, a fully autonomous pipeline that procedurally generates terminal-use tasks without human annotation. The pipeline has four stages: generating diverse task descriptions, building and validating containerized environments, producing completion tests, and filtering for solvability. From this pipeline we obtain 3255 tasks spanning file operations, log management, data processing, scripting, and database operations. We train agents using vanilla PPO with binary episode level rewards and a minimal interaction loop: no retrieval, multi-agent coordination, or specialized tools. Despite this simplicity, models trained on Endless Terminals show substantial gains: on our held-out dev set, Llama-3.2-3B improves from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%. These improvements transfer to human-curated benchmarks: models trained on Endless Terminals show substantial gains on held out human curated benchmarks: on TerminalBench 2.0, Llama-3.2-3B improves from 0.0% to 2.2%, Qwen2.5-7B from 2.2% to 3.4%, and Qwen3-8B-openthinker-sft from 1.1% to 6.7%, in each case outperforming alternative approaches including models with more complex agentic scaffolds. These results demonstrate that simple RL succeeds when environments scale.