# WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG

arXiv: https://arxiv.org/abs/2603.23497 · HTML: https://mlsanity.com/paper/2603.23497/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 43 of its 53 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)
- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation](https://mlsanity.com/paper/2407.02371/) (2024)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [LongLive: Real-time Interactive Long Video Generation](https://mlsanity.com/paper/2509.22622/) (2025)
- [LTX-2: Efficient Joint Audio-Visual Foundation Model](https://mlsanity.com/paper/2601.03233/) (2026)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [WorldScore: A Unified Evaluation Benchmark for World Generation](https://mlsanity.com/paper/2504.00983/) (2025)
- [Advancing Open-source World Models](https://mlsanity.com/paper/2601.20540/) (2026)
- [MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions](https://mlsanity.com/paper/2407.06358/) (2024)
- [GameFactory: Creating New Games with Generative Interactive Videos](https://mlsanity.com/paper/2501.08325/) (2025)
- [GameGen-X: Interactive Open-world Game Video Generation](https://mlsanity.com/paper/2411.00769/) (2024)
- [WorldModelBench: Judging Video Generation Models As World Models](https://mlsanity.com/paper/2502.20694/) (2025)
- [Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation](https://mlsanity.com/paper/2506.04225/) (2025)
- [Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition](https://mlsanity.com/paper/2506.17201/) (2025)
- [T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback](https://mlsanity.com/paper/2405.18750/) (2024)
- [Yume-1.5: A Text-Controlled Interactive World Generation Model](https://mlsanity.com/paper/2512.22096/) (2025)
- [Yume: An Interactive World Generation Model](https://mlsanity.com/paper/2507.17744/) (2025)

## Built on by (most cited first, 13 of 13)

- [WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models](https://mlsanity.com/paper/2606.31672/) (2026)
- [The DAWN of World-Action Interactive Models](https://mlsanity.com/paper/2605.11550/) (2026)
- [StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation](https://mlsanity.com/paper/2607.26754/) (2026)
- [Marionette: Predicting World States, Rendering Geometry, Painting Appearance](https://mlsanity.com/paper/2608.14530/) (2026)
- [From Pixels to States: Rethinking Interactive World Models as Game Engines](https://mlsanity.com/paper/2607.14076/) (2026)
- [World Models as Group Actions](https://mlsanity.com/paper/2605.24578/) (2026)
- [WorldMind: Decoupled Game World Model for State-Aware NPC Behavior](https://mlsanity.com/paper/2608.21439/) (2026)
- [AI for Games in the Foundation Model Era](https://mlsanity.com/paper/2609.16679/) (2026)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations](https://mlsanity.com/paper/2608.15659/) (2026)
- [HelloWorld: Enabling Socially Interactive Characters in Video World Models](https://mlsanity.com/paper/2608.05070/) (2026)
- [GameDirector: Decoupling Gameplay Logic from Rendering for Player-Configurable Game World Models](https://mlsanity.com/paper/2609.25652/) (2026)
- [GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay](https://mlsanity.com/paper/2609.25001/) (2026)

## Abstract (arXiv)

Dynamical systems theory and reinforcement learning view world evolution as latent-state dynamics driven by actions, with visual observations providing partial information about the state. Recent video world models attempt to learn this action-conditioned dynamics from data. However, existing datasets rarely match the requirement: they typically lack diverse and semantically meaningful action spaces, and actions are directly tied to visual observations rather than mediated by underlying states. As a result, actions are often entangled with pixel-level changes, making it difficult for models to learn structured world dynamics and maintain consistent evolution over long horizons. In this paper, we propose WildWorld, a large-scale action-conditioned world modeling dataset with explicit state annotations, automatically collected from a photorealistic AAA action role-playing game (Monster Hunter: Wilds). WildWorld contains over 108 million frames and features more than 450 actions, including movement, attacks, and skill casting, together with synchronized per-frame annotations of character skeletons, world states, camera poses, and depth maps. We further derive WildBench to evaluate models through Action Following and State Alignment. Extensive experiments reveal persistent challenges in modeling semantically rich actions and maintaining long-horizon state consistency, highlighting the need for state-aware video generation. The project page is https://shandaai.github.io/wildworld-project/.