mlsanityData to 7 Oct 2026

VideoWorld: Exploring Knowledge Learning from Unlabeled Videos

arXiv 2501.09781 ↗First posted 16 Jan 2025cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

This work explores whether a deep generative model can learn complex knowledge solely from visual input, in contrast to the prevalent focus on text-based models like large language models (LLMs). We develop VideoWorld, an auto-regressive video generation model trained on unlabeled video data, and test its knowledge acquisition abilities in video-based Go and robotic control tasks. Our experiments reveal two key findings: (1) video-only training provides sufficient information for learning knowledge, including rules, reasoning and planning capabilities, and (2) the representation of visual change is crucial for knowledge acquisition. To improve both the efficiency and efficacy of this process, we introduce the Latent Dynamics Model (LDM) as a key component of VideoWorld. Remarkably, VideoWorld reaches a 5-dan professional level in the Video-GoBench with just a 300-million-parameter model, without relying on search algorithms or reward mechanisms typical in reinforcement learning. In robotic tasks, VideoWorld effectively learns diverse control operations and generalizes across environments, approaching the performance of oracle models in CALVIN and RLBench. This study opens new avenues for knowledge acquisition from visual data, with all code, data, and models open-sourced for further research.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 57 of its 76 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

57 of 76 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  2. 2023Mistral 7Bcited by 3,065
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  8. 2024Genie: Generative Interactive Environmentscited by 706
  9. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  10. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  11. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  12. 2023Finite Scalar Quantization: VQ-VAE Made Simplecited by 425
  13. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  14. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  15. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  16. 2022MAGVIT: Masked Generative Video Transformercited by 341
  17. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  18. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  19. 2019Learning Latent Plans from Playcited by 310
  20. 2024Latent Action Pretraining from Videoscited by 302
  21. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  22. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  23. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  24. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  25. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  26. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  27. 2023Learning to Act without Actionscited by 116
  28. 2024Video as the New Language for Real-World Decision Makingcited by 86
  29. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 73
  30. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  31. 2019Accelerating Self-Play Learning in Gocited by 60
  32. 2024Determinants of LLM-assisted Decision-Makingcited by 55
  33. 2024ChatVTG: Video Temporal Grounding via Chat with Video Dialogue Large Language Modelscited by 54
  34. 2024Context-Guided Spatio-Temporal Video Groundingcited by 45
  35. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 45
  36. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learningcited by 33
  37. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 27
  38. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 25; 2026: 17.

42 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  3. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  4. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  5. 2025VGR: Visual Grounded Reasoningcited by 70
  6. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 48
  7. 2026DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulationcited by 33
  8. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  9. 2025BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanationcited by 30
  10. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  11. 2025FlexVAR: Flexible Visual Autoregressive Modeling without Residual Predictioncited by 22
  12. 2025MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglementcited by 22
30 more
  1. 2025CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidancecited by 19
  2. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  3. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 14
  4. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 11
  5. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 11
  6. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 10
  7. 2025A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysiscited by 9
  8. 2025Video-GPT via Next Clip Diffusioncited by 8
  9. 2025PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusioncited by 7
  10. 2025The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Textcited by 7
  11. 2026Learning to Act Robustly with View-Invariant Latent Actionscited by 6
  12. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  13. 2026GeoWorld: Geometric World Modelscited by 6
  14. 2026Learning Additively Compositional Latent Actions for Embodied AIcited by 6
  15. 2026Orca: The World is in Your Mindcited by 6
  16. 2026MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestrationcited by 5
  17. 2025Terra: Explorable Native 3D World Model with Point Latentscited by 4
  18. 2026iFSQ: Improving FSQ for Image Generation with 1 Line of Codecited by 4
  19. 2026LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generationcited by 4
  20. 2025A Survey: Spatiotemporal Consistency in Video Generationcited by 3
  21. 2025LaVieID: Local Autoregressive Diffusion Transformers for Identity-Preserving Video Creationcited by 3
  22. 2025Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistencycited by 2
  23. 2026PhiZero: A World Model Built Around Physical Languagecited by 2
  24. 2025Seeing Space and Motion: Enhancing Latent Actions with Geometric and Dynamic Awareness for Vision-Language-Action Modelscited by 1
  25. 2025Learning to play: A Multimodal Agent for 3D Game-Playcited by 1
  26. 2025In-Video Instructions: Visual Signals as Generative Controlcited by 1
  27. 2026P-Flow: Prompting Visual Effects Generationcited by 0
  28. 2026Predicting Consequences and Reinforcing Navigation Policies with Latent World Modelscited by 0
  29. 2026AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agentscited by 0
  30. 2026IntactWorld: Joint World Modeling with Intact Featurescited by 0