mlsanityData to 14 Oct 2026

iVideoGPT: Interactive VideoGPTs are Scalable World Models

arXiv 2405.15223 ↗First posted 24 May 2024cs.CV cs.LG cs.ROcited by 146

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

World models empower model-based agents to interactively explore, reason, and plan within imagined environments for real-world decision-making. However, the high demand for interactivity poses challenges in harnessing recent advancements in video generative models for developing world models at scale. This work introduces Interactive VideoGPT (iVideoGPT), a scalable autoregressive transformer framework that integrates multimodal signals--visual observations, actions, and rewards--into a sequence of tokens, facilitating an interactive experience of agents via next-token prediction. iVideoGPT features a novel compressive tokenization technique that efficiently discretizes high-dimensional visual observations. Leveraging its scalable architecture, we are able to pre-train iVideoGPT on millions of human and robotic manipulation trajectories, establishing a versatile foundation that is adaptable to serve as interactive world models for a wide range of downstream tasks. These include action-conditioned video prediction, visual planning, and model-based reinforcement learning, where iVideoGPT achieves competitive performance compared with state-of-the-art methods. Our work advances the development of interactive general world models, bridging the gap between generative video models and practical model-based reinforcement learning applications. Code and pre-trained models are available at https://thuml.github.io/iVideoGPT.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 95 of its 112 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

95 of 112 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Structured World Models from Human Videos21 Aug
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  3. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning4 Feb

Other references · 92

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
38 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2022Video Diffusion Modelscited by 2,305
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  12. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,436
  13. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  14. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  15. 2023Mastering Diverse Domains through World Modelscited by 1,253
  16. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  17. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  18. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  19. 2020Mastering Atari with Discrete World Modelscited by 1,007
  20. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  21. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  22. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  23. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  24. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  25. 2024Genie: Generative Interactive Environmentscited by 747
  26. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  27. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  28. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  29. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  30. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  31. 2019Model-Based Reinforcement Learning for Ataricited by 502
  32. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  33. 2023Learning Interactive Real-World Simulatorscited by 458
  34. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  35. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  36. 2022MAGVIT: Masked Generative Video Transformercited by 346
  37. 2022Transformers are Sample-Efficient World Modelscited by 328
  38. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 62; 2026: 71.

146 papers.

Sort:

By area:

  1. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  2. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  3. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  4. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
  5. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  6. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  7. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  8. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraftcited by 97
  9. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 76
  10. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 74
  11. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 72
  12. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
38 more
  1. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  2. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 67
  3. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 52
  4. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 52
  5. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  7. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  8. 2025Pre-Trained Video Generative Models as World Simulatorscited by 44
  9. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 42
  10. 2026VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Modelcited by 41
  11. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 40
  12. 2025RoboScape: Physics-informed Embodied World Modelcited by 40
  13. 2024Autoregressive Models in Vision: A Surveycited by 39
  14. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 39
  15. 2025VRAG: Learning World Models for Interactive Video Generationcited by 38
  16. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 37
  17. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  18. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulationcited by 35
  19. 2025Astra: General Interactive World Model with Autoregressive Denoisingcited by 33
  20. 2024SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generationcited by 28
  21. 2025Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environmentscited by 28
  22. 2025Playing with Transformer at 30+ FPS via Next-Frame Diffusioncited by 20
  23. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 20
  24. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  25. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 19
  26. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  27. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  28. 2025Graph World Modelcited by 16
  29. 2026LIVE: Long-horizon Interactive Video World Modelingcited by 16
  30. 2025Goal-VLA: Image-Generative VLMs as Object-Centric World Models Empowering Zero-shot Robot Manipulationcited by 15
  31. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  32. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  33. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  34. 2024VidTwin: Video VAE with Decoupled Structure and Dynamicscited by 14
  35. 2026World-Value-Action Model: Implicit Planning for Vision-Language-Action Systemscited by 14
  36. 2026Code World Model: Coding Agent as World Braincited by 14
  37. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
  38. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 13

50 most cited of 146 All 146 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Interactive World Simulator for Robot Policy Training and Evaluation
  2. 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
  3. 2026WALL-SS: Scaling Long-horizon World Models via Next-Scale Autoregression
  4. 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players
  5. 2026InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation
  6. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learning
  7. 2026ω-EVA: Envision, Verify, and Act with Latent Interactive World Models
  8. 2026PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration