# WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning

arXiv: https://arxiv.org/abs/2506.04363 · HTML: https://mlsanity.com/paper/2506.04363/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and robot world models, one of 11 directions in World models.
- Builds on 57 of its 63 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- World models / [Embodied and robot world models](https://mlsanity.com/wm/embodied-and-robot-world-models/)

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [GAIA-1: A Generative World Model for Autonomous Driving](https://mlsanity.com/paper/2309.17080/) (2023)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Revisiting Feature Prediction for Learning Visual Representations from Video](https://mlsanity.com/paper/2404.08471/) (2024)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change](https://mlsanity.com/paper/2206.10498/) (2022)
- [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023)
- [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](https://mlsanity.com/paper/2311.17918/) (2023)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Navigation World Models](https://mlsanity.com/paper/2412.03572/) (2024)
- [Interactive Language: Talking to Robots in Real Time](https://mlsanity.com/paper/2210.06407/) (2022)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [RoboDreamer: Learning Compositional World Models for Robot Imagination](https://mlsanity.com/paper/2404.12377/) (2024)
- [WorldScore: A Unified Evaluation Benchmark for World Generation](https://mlsanity.com/paper/2504.00983/) (2025)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)

## Built on by (most cited first, 20 of 20)

- [VL-JEPA: Joint Embedding Predictive Architecture for Vision-language](https://mlsanity.com/paper/2512.10942/) (2025)
- [Embodied AI Agents: Modeling the World](https://mlsanity.com/paper/2506.22355/) (2025)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [MobileWorldBench: Towards Semantic World Modeling For Mobile Agents](https://mlsanity.com/paper/2512.14014/) (2025)
- [ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction](https://mlsanity.com/paper/2511.20937/) (2025)
- [Environment Scaling for Interactive Agentic Experience Collection: A Survey](https://mlsanity.com/paper/2511.09586/) (2025)
- [Action100M: A Large-scale Video Action Dataset](https://mlsanity.com/paper/2601.10592/) (2026)
- [PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models](https://mlsanity.com/paper/2601.16007/) (2026)
- [How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Position](https://mlsanity.com/paper/2606.15032/) (2026)
- [Self-Improving World Modelling with Latent Actions](https://mlsanity.com/paper/2602.06130/) (2026)
- [Bridging the Agent-World Gap: Text World Models for LLM-based Agents](https://mlsanity.com/paper/2606.09032/) (2026)
- [A Definition and Roadmap for World Models](https://mlsanity.com/paper/2607.06401/) (2026)
- [ChronoDreamer: Action-Conditioned World Model as an Online Simulator for Robotic Planning](https://mlsanity.com/paper/2512.18619/) (2025)
- [SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios](https://mlsanity.com/paper/2511.17649/) (2025)
- [Modeling the Mental World for Embodied AI: A Comprehensive Review](https://mlsanity.com/paper/2601.02378/) (2025)
- [World-Model Collapse as a Phase Transition](https://mlsanity.com/paper/2606.31399/) (2026)
- [Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence](https://mlsanity.com/paper/2609.29669/) (2026)
- [VIDEAS: Distilling Explicit Action Semantics from Demonstration Videos for World Models via Prior-Guided Simulation](https://mlsanity.com/paper/2609.33464/) (2026)
- [WOVEN: Weaving Visual World Modeling into Multimodal LLMs](https://mlsanity.com/paper/2610.12417/) (2026)
- [CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox Worlds](https://mlsanity.com/paper/2610.16450/) (2026)

## Abstract (arXiv)

Humans are known to have an internal "world model" that enables us to carry out action planning based on world states. AI agents need to have such a world model for action planning as well. It is not clear how current AI models, especially generative models, are able to learn such world models and carry out procedural planning in diverse environments. We introduce WorldPrediction, a video-based benchmark for evaluating world modeling and procedural planning capabilities of different AI models. In contrast to prior benchmarks that focus primarily on low-level world modeling and robotic motion planning, WorldPrediction is the first benchmark that emphasizes actions with temporal and semantic abstraction. Given initial and final world states, the task is to distinguish the proper action (WorldPrediction-WM) or the properly ordered sequence of actions (WorldPrediction-PP) from a set of counterfactual distractors. This discriminative task setup enable us to evaluate different types of world models and planners and realize a thorough comparison across different hypothesis. The benchmark represents states and actions using visual observations. In order to prevent models from exploiting low-level continuity cues in background scenes, we provide "action equivalents" - identical actions observed in different contexts - as candidates for selection. This benchmark is grounded in a formal framework of partially observable semi-MDP, ensuring better reliability and robustness of the evaluation. We conduct extensive human filtering and validation on our benchmark and show that current frontier models barely achieve 57% accuracy on WorldPrediction-WM and 38% on WorldPrediction-PP whereas humans are able to solve both tasks perfectly.