mlsanityData to 14 Oct 2026

PEEKABOO: Interactive Video Generation via Masked-Diffusion

arXiv 2312.07509 ↗First posted 12 Dec 2023cs.CV cs.LGcited by 76

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

Modern video generation models like Sora have achieved remarkable success in producing high-quality videos. However, a significant limitation is their inability to offer interactive control to users, a feature that promises to open up unprecedented applications and creativity. In this work, we introduce the first solution to equip diffusion-based video generation models with spatio-temporal control. We present Peekaboo, a novel masked attention module, which seamlessly integrates with current video generation models offering control without the need for additional training or inference overhead. To facilitate future research, we also introduce a comprehensive benchmark for interactive video generation. This benchmark offers a standardized framework for the community to assess the efficacy of emerging interactive video generation models. Our extensive qualitative and quantitative assessments reveal that Peekaboo achieves up to a 3.8x improvement in mIoU over baseline models, all while maintaining the same latency. Code and benchmark are available on the webpage.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 30 of its 40 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

30 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb

Other references · 29

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  5. 2022Video Diffusion Modelscited by 2,305
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  7. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  8. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  9. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
17 more
  1. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  2. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  3. 2023ModelScope Text-to-Video Technical Reportcited by 658
  4. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  6. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  7. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  8. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  9. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  10. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 164
  11. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  12. 2020LaSOT: A High-quality Large-scale Single Object Tracking Benchmarkcited by 132
  13. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  14. 2023LLM-grounded Video Diffusion Modelscited by 83
  15. 2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesiscited by 78
  16. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  17. 2018On the effectiveness of task granularity for transfer learningcited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 25; 2025: 29; 2026: 21.

76 papers.

Sort:

By area:

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  2. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  3. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  4. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  5. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  6. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120
  7. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  8. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  9. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
  10. 2024Learning Temporally Consistent Video Depth from Video Diffusion Priorscited by 100
  11. 2025Controllable Video Generation: A Surveycited by 98
  12. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 93
38 more
  1. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 91
  2. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  3. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  4. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 61
  5. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 57
  6. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 56
  7. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  8. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 43
  9. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 43
  10. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 35
  11. 2025MotionPro: A Precise Motion Controller for Image-to-Video Generationcited by 30
  12. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  13. 2024I2VControl: Disentangled and Unified Video Motion Synthesis Controlcited by 20
  14. 2024Compositional 3D-aware Video Generation with LLM Directorcited by 18
  15. 2024ObjCtrl-2.5D: Training-free Object Control with Camera Posescited by 18
  16. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  17. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  18. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  19. 2024Motion Modes: What Could Happen Next?cited by 12
  20. 2025DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generationcited by 11
  21. 2025We'll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedbackcited by 10
  22. 2025DisMo: Disentangled Motion Representations for Open-World Motion Transfercited by 10
  23. 2026SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generationcited by 10
  24. 2024Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verificationcited by 9
  25. 2024Mojito: Motion Trajectory and Intensity Control for Video Generationcited by 9
  26. 2025Consistent Video Editing as Flow-Driven Image-to-Video Generationcited by 8
  27. 2025CamPVG: Camera-Controlled Panoramic Video Generation with Epipolar-Aware Diffusioncited by 8
  28. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  29. 2024CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusioncited by 7
  30. 2024DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptationcited by 7
  31. 2025Target-Aware Video Diffusion Modelscited by 7
  32. 2025LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusioncited by 7
  33. 2025ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignmentcited by 7
  34. 2026HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesiscited by 7
  35. 2026Evolution of Video Generative Foundationscited by 7
  36. 2024Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversioncited by 6
  37. 2025DreamInsert: Zero-Shot Image-to-Video Object Insertion from A Single Imagecited by 6
  38. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videoscited by 6

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  2. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation
  3. 2026Efficient Video Diffusion Models: Advancements and Challenges
  4. 2025Resource-Efficient Motion Control for Video Generation via Dynamic Mask Guidance
  5. 2024Video Diffusion Models: A Survey
  6. 2024MarDini: Masked Autoregressive Diffusion for Video Generation at Scale
  7. 2023MoVideo: Motion-Aware Video Generation with Diffusion Models
  8. 2026UniVid: Pyramid Diffusion Model for High Quality Video Generation