# ECHO: Terminal Agents Learn World Models for Free

arXiv: https://arxiv.org/abs/2605.24517 · HTML: https://mlsanity.com/paper/2605.24517/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 23 May 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 18 of 18)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.02479/) (2025)
- [OpenClaw-RL: Train Any Agent Simply by Talking](https://mlsanity.com/paper/2603.10165/) (2026)
- [Endless Terminals: Scaling RL Environments for Terminal Agents](https://mlsanity.com/paper/2601.16443/) (2026)
- [Expanding the Capabilities of Reinforcement Learning via Text Feedback](https://mlsanity.com/paper/2602.02482/) (2026)
- [SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent](https://mlsanity.com/paper/2511.16108/) (2025)
- [World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning](https://mlsanity.com/paper/2503.10480/) (2025)
- [An Empirical Study on the Power of Future Prediction in Partially Observable Environments](https://mlsanity.com/paper/2402.07102/) (2024)

## Built on by (most cited first, 19 of 19)

- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and The Channel, Not the Content, Decides What Works](https://mlsanity.com/paper/2607.21273/) (2026)
- [Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL](https://mlsanity.com/paper/2606.14211/) (2026)
- [Antares: Foundation Models for Agentic Vulnerability Localization](https://mlsanity.com/paper/2608.02407/) (2026)
- [AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL](https://mlsanity.com/paper/2608.24114/) (2026)
- [Terminal Agents: A Survey of AI Agents in Command-Line Environments](https://mlsanity.com/paper/2608.20485/) (2026)
- [RLTL;DR: Self-improvement by Internalizing Self-generated Feedback](https://mlsanity.com/paper/2609.37633/) (2026)
- [AgentOmnia: Scaling Agentic Models for Full-Scenario Applications](https://mlsanity.com/paper/2607.23124/) (2026)
- [How do World Models and Policies Compose in LLM Agents? A Joint Spectral and Behavioral Account](https://mlsanity.com/paper/2608.30067/) (2026)
- [Don't Mask the Environment: Observation Supervision Changes How Agents Explore Under RL](https://mlsanity.com/paper/2609.20715/) (2026)
- [Agent-Editing World Model: Rethinking World Modeling for LLM Agents](https://mlsanity.com/paper/2609.28416/) (2026)
- [Does Learning to Predict the World Help Agents Act? Auditing World-Model Post-Training](https://mlsanity.com/paper/2609.33335/) (2026)
- [Selecting Long-Horizon Trajectories for Reliable and Efficient Terminal-Agent Training](https://mlsanity.com/paper/2610.05831/) (2026)
- [Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation](https://mlsanity.com/paper/2610.11384/) (2026)
- [ObsSpec: Accelerating Agentic Reinforcement Learning with Observation Speculation](https://mlsanity.com/paper/2610.15246/) (2026)
- [ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs](https://mlsanity.com/paper/2610.15368/) (2026)
- [ReTerm: Recovery Training for Self-Improving Terminal Agents](https://mlsanity.com/paper/2610.18484/) (2026)
- [Model What Matters: Decision-Guided Selective World Modeling for Language Agents](https://mlsanity.com/paper/2610.19642/) (2026)
- [Co-Evolving Acting and Forecasting for Language Agents](https://mlsanity.com/paper/2610.20597/) (2026)

## Abstract (arXiv)

CLI agents are the closest thing language models have to an embodied setting: the model emits commands, the terminal executes them, and the returned stream -- stdout, errors, files, logs, and traces -- records the consequences. We argue that this stream is a supervision signal, but standard agent RL discards it: GRPO-style training updates action tokens with sparse outcome-level rewards while ignoring environment responses already in the rollout. Failed rollouts provide little policy-gradient signal despite containing rich evidence about how the environment responds. We introduce ECHO (Environment Cross-entropy Hybrid Objective), a hybrid objective that combines the standard policy-gradient loss on action tokens with an auxiliary loss that trains the policy to predict environment observation tokens resulting from its own actions. ECHO reuses the same forward pass as GRPO, requires no additional rollouts, and turns terminal feedback into dense supervision for all rollouts. ECHO doubles GRPO pass@1 on TerminalBench-2.0: Qwen3-8B improves from 2.70% to 5.17%, and Qwen3-14B from 5.17% to 10.79%. ECHO also produces policies that better predict terminal dynamics, even on trajectories they did not generate: across held-out rollouts, it sharply reduces environment-token cross-entropy while GRPO alone barely changes it. From base Qwen3-8B, ECHO matches expert-SFT-then-GRPO performance on held-out terminal tasks without expert demonstrations, and recovers roughly half of the expert-SFT initialization benefit on TerminalBench-2.0. In some settings, the environment prediction loss alone enables verifier-free self-improvement, allowing policies to improve on unseen OOD tasks by learning only from environment interactions. Together, these results suggest that environment observations are not merely context for future actions, but a dense, on-policy supervision signal already present in every rollout.