mlsanityData to 14 Oct 2026

Seer: Language Instructed Video Prediction with Latent Diffusion Models

arXiv 2303.14897 ↗First posted 27 Mar 2023cs.CVcited by 52

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Imagining the future trajectory is the key for robots to make sound planning and successfully reach their goals. Therefore, text-conditioned video prediction (TVP) is an essential task to facilitate general robot policy learning. To tackle this task and empower robots with the ability to foresee the future, we propose a sample and computation-efficient model, named Seer, by inflating the pretrained text-to-image (T2I) stable diffusion models along the temporal axis. We enhance the U-Net and language conditioning model by incorporating computation-efficient spatial-temporal attention. Furthermore, we introduce a novel Frame Sequential Text Decomposer module that dissects a sentence's global instruction into temporally aligned sub-instructions, ensuring precise integration into each frame of generation. Our framework allows us to effectively leverage the extensive prior knowledge embedded in pretrained T2I models across the frames. With the adaptable-designed architecture, Seer makes it possible to generate high-fidelity, coherent, and instruction-aligned video frames by fine-tuning a few layers on a small amount of data. The experimental results on Something Something V2 (SSv2), Bridgedata and EpicKitchens-100 datasets demonstrate our superior video prediction performance with around 480-GPU hours versus CogVideo with over 12,480-GPU hours: achieving the 31% FVD improvement compared to the current SOTA model on SSv2 and 83.7% average preference in the human evaluation.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 48 of its 59 references that are in the map. First posted on arXiv 27 Mar 2023.

Builds on

48 of 59 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2017Neural Discrete Representation Learningcited by 4,712
36 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2022Video Diffusion Modelscited by 2,305
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  13. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  14. 2016Generative Adversarial Text to Image Synthesiscited by 735
  15. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  16. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  17. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  18. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  19. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  20. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  21. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  22. 2022MAGVIT: Masked Generative Video Transformercited by 346
  23. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  24. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  25. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  26. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  27. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  28. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
  29. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  30. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  31. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  32. 2022SimVP: Simpler yet Better Video Predictioncited by 184
  33. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  34. 2022Diffusion Models for Video Prediction and Infillingcited by 135
  35. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  36. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 18; 2025: 21; 2026: 6.

52 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  5. 2023A Survey on Video Diffusion Modelscited by 206
  6. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  7. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 85
  8. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  9. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 68
  10. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  11. 2023Generative Semantic Communication: Diffusion Models Beyond Bit Recoverycited by 41
  12. 2024TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Modelscited by 31
38 more
  1. 2024AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AIcited by 27
  2. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 21
  3. 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Owncited by 19
  4. 2025EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videoscited by 19
  5. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  6. 2024AniClipart: Clipart Animation with Text-to-Video Priorscited by 17
  7. 2025Object-Centric World Model for Language-Guided Manipulationcited by 16
  8. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  9. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  10. 2024UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Controlcited by 14
  11. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  12. 2025Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Thinkcited by 12
  13. 2024GenRec: Unifying Video Generation and Recognition with Diffusion Modelscited by 11
  14. 2024PoseCrafter: One-Shot Personalized Video Synthesis Following Flexible Pose Controlcited by 10
  15. 2025A Physical Coherence Benchmark for Evaluating Video Generation Models via Optical Flow-guided Frame Predictioncited by 10
  16. 2025Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Reviewcited by 10
  17. 2026An Efficient and Multi-Modal Navigation System with One-Step World Modelcited by 9
  18. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  19. 2024TIV-Diffusion: Towards Object-Centric Movement for Text-driven Image to Video Generationcited by 7
  20. 2025Pixel Motion Diffusion is What We Need for Robot Controlcited by 7
  21. 2025Robotic VLA Benefits from Joint Learning with Motion Image Diffusioncited by 7
  22. 2025STORM: Search-Guided Generative World Models for Robotic Manipulationcited by 7
  23. 2026Future Optical Flow Prediction Improves Robot Control & Video Generationcited by 7
  24. 2025VILP: Imitation Learning with Latent Video Planningcited by 6
  25. 2024Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generationcited by 5
  26. 2026Forecasting Motion in the Wildcited by 5
  27. 2024Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicalitycited by 4
  28. 2025AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Modelscited by 4
  29. 2025Frozen Forecasting: A Unified Evaluationcited by 4
  30. 2025Trajectory Conditioned Cross-embodiment Skill Transfercited by 3
  31. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  32. 2025Frame-wise Conditioning Adaptation for Fine-Tuning Diffusion Models in Text-to-Video Predictioncited by 2
  33. 2025EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generationcited by 2
  34. 2025Show Me: Unifying Instructional Image and Video Generation with Diffusion Modelscited by 2
  35. 2025TextOCVP: Object-Centric Video Prediction with Language Guidancecited by 1
  36. 2025Learning to Animate Images from A Few Videos to Portray Delicate Human Actionscited by 1
  37. 2026Instruction-Driven 3D Facial Expression Generation and Transitioncited by 1
  38. 2025PoseDiff: A Unified Diffusion Model Bridging Robot Pose Estimation and Video-to-Action Controlcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Causal World Modeling for Robot Control
  2. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  3. 2023Video Language Planning
  4. 2025Large Video Planner Enables Generalizable Robot Control
  5. 2026DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
  6. 2025Image Generation as a Visual Planner for Robotic Manipulation
  7. 2026Native Video-Action Pretraining for Generalizable Robot Control
  8. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations