# Imagine-then-Plan: Agent Learning from Adaptive Lookahead with World Models

arXiv: https://arxiv.org/abs/2601.08955 · HTML: https://mlsanity.com/paper/2601.08955/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 13 Jan 2026.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 26 of 26)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [FireAct: Toward Language Agent Fine-tuning](https://mlsanity.com/paper/2310.05915/) (2023)
- [Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning](https://mlsanity.com/paper/2305.14909/) (2023)
- [Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation](https://mlsanity.com/paper/2203.16487/) (2022)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement](https://mlsanity.com/paper/2406.11176/) (2024)
- [Agent Learning via Early Experience](https://mlsanity.com/paper/2510.08558/) (2025)
- [Agent-R: Training Language Model Agents to Reflect via Iterative Self-Training](https://mlsanity.com/paper/2501.11425/) (2025)
- [SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution](https://mlsanity.com/paper/2505.20732/) (2025)
- [Agent Planning with World Knowledge Model](https://mlsanity.com/paper/2405.14205/) (2024)
- [VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents](https://mlsanity.com/paper/2510.16907/) (2025)
- [AgentRefine: Enhancing Agent Generalization through Refinement Tuning](https://mlsanity.com/paper/2501.01702/) (2025)
- [From Word to World: Can Large Language Models be Implicit Text-based World Models?](https://mlsanity.com/paper/2512.18832/) (2025)
- [STeCa: Step-level Trajectory Calibration for LLM Agent Learning](https://mlsanity.com/paper/2502.14276/) (2025)
- [Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective](https://mlsanity.com/paper/2510.15047/) (2025)
- [World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning](https://mlsanity.com/paper/2503.10480/) (2025)
- [DMWM: Dual-Mind World Model with Long-Term Imagination](https://mlsanity.com/paper/2502.07591/) (2025)

## Built on by (most cited first, 8 of 8)

- [Qwen-AgentWorld: Language World Models for General Agents](https://mlsanity.com/paper/2606.24597/) (2026)
- [Policy and World Modeling Co-Training for Language Agents](https://mlsanity.com/paper/2606.02388/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Retrieval-Augmented LLM Agents: Learning to Learn from Experience](https://mlsanity.com/paper/2603.18272/) (2026)
- [MemWM: Memory-Augmented Text-Based World Model](https://mlsanity.com/paper/2608.07107/) (2026)
- [HyperWorld: Hypergraph-Structured State Serialization Improves Learned Textual World Models](https://mlsanity.com/paper/2609.00002/) (2026)
- [EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Making](https://mlsanity.com/paper/2609.38334/) (2026)
- [EvoBelief: Falsifiable World Models for Self-Improving LLM Agents](https://mlsanity.com/paper/2610.12789/) (2026)

## Abstract (arXiv)

Recent advances in world models have shown promise for modeling future dynamics of environmental states, enabling agents to reason and act without accessing real environments. Current methods mainly perform single-step or fixed-horizon rollouts, leaving their potential for complex task planning under-exploited. We propose Imagine-then-Plan (textttITP), a unified framework for agent learning via lookahead imagination, where an agent's policy model interacts with the learned world model, yielding multi-step ``imagined'' trajectories. Since the imagination horizon may vary by tasks and stages, we introduce a novel adaptive lookahead mechanism by trading off the ultimate goal and task progress. The resulting imagined trajectories provide rich signals about future consequences, such as achieved progress and potential conflicts, which are fused with current observations, formulating a partially textitobservable and textitimaginable Markov decision process to guide policy learning. We instantiate textttITP with both training-free and reinforcement-trained variants. Extensive experiments across representative agent benchmarks demonstrate that textttITP significantly outperforms competitive baselines. Further analyses validate that our adaptive lookahead largely enhances agents' reasoning capability, providing valuable insights into addressing broader, complex tasks. Our code and data will be publicly available at https://github.com/loyiv/ITP.