# EgoCS-400K: An Egocentric Gameplay Dataset for World Models arXiv: https://arxiv.org/abs/2606.18180 · HTML: https://mlsanity.com/paper/2606.18180/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Video-generation world models and simulators, one of 11 directions in World models. - Builds on 40 of its 47 references that are in the map. - First posted on arXiv 16 Jun 2026. ## Placed in - World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/) ## Builds on (most cited first, 30 of 40) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019) - [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017) - [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021) - [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018) - [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021) - [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017) - [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019) - [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024) - [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024) - [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018) - [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024) - [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024) - [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017) - [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020) - [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022) - [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024) - [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024) - [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019) - [WorldMem: Long-term Consistent World Simulation with Memory](https://mlsanity.com/paper/2504.12369/) (2025) - [GameFactory: Creating New Games with Generative Interactive Videos](https://mlsanity.com/paper/2501.08325/) (2025) - [Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation](https://mlsanity.com/paper/2506.04225/) (2025) - [Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory](https://mlsanity.com/paper/2604.08995/) (2026) - [Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition](https://mlsanity.com/paper/2506.17201/) (2025) ## Built on by (most cited first, 5 of 5) - [Marionette: Predicting World States, Rendering Geometry, Painting Appearance](https://mlsanity.com/paper/2608.14530/) (2026) - [From Pixels to States: Rethinking Interactive World Models as Game Engines](https://mlsanity.com/paper/2607.14076/) (2026) - [WorldMind: Decoupled Game World Model for State-Aware NPC Behavior](https://mlsanity.com/paper/2608.21439/) (2026) - [GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay](https://mlsanity.com/paper/2609.25001/) (2026) - [Esports World Model Dataset](https://mlsanity.com/paper/2610.16166/) (2026) ## Abstract (arXiv) The shift from video generation to interactive world modeling places new demands on data: beyond captioned videos, world models require temporally aligned video-action-language trajectories grounded in the actions, camera motion, states, and events that drive future scene changes. However, such data is difficult to obtain at scale. Web video datasets offer broad visual coverage but lack executable actions and reliable states; robotic datasets provide action and state supervision but are costly and limited in scene diversity; and existing simulators often lack large-scale human-driven interaction trajectories. In this paper, we introduce EgoCS-400K, a large-scale replay-grounded egocentric Counter-Strike dataset for world models, built from public professional CS and CS2 match demos that preserve human gameplay trajectories and enable parsing, replaying, rendering, and temporal alignment. We extract player states, view directions, movements, keyboard/button inputs, view-angle changes, weapon usage, game events, and round-level context, and render clean first-person videos from the same trajectories. EgoCS-400K contains over 400,000 first-person videos and 10,000 hours of gameplay from more than 1,000 matches and 40,000 rounds, covering 13 maps and 10 player viewpoints per round. It supports a range of interactive visual modeling tasks, including action-conditioned future prediction, state- and event-aware scene rollout, replay-grounded captioning, and agent egocentric action understanding. By connecting visual observations with human actions, camera motion, game states, and events at scale, EgoCS-400K serves as a practical bridge between passive web videos, controllable game simulation, and costly real-world embodied data.