mlsanityData to 7 Oct 2026

Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation

arXiv 2304.08477 ↗First posted 17 Apr 2023cs.CVcited by 137

On the map: Diffusion models → Video generation

Abstract (arXiv)

We propose Latent-Shift -- an efficient text-to-video generation method based on a pretrained text-to-image generation model that consists of an autoencoder and a U-Net diffusion model. Learning a video diffusion model in the latent space is much more efficient than in the pixel space. The latter is often limited to first generating a low-resolution video followed by a sequence of frame interpolation and super-resolution models, which makes the entire pipeline very complex and computationally expensive. To extend a U-Net from image generation to video generation, prior work proposes to add additional modules like 1D temporal convolution and/or temporal attention layers. In contrast, we propose a parameter-free temporal shift module that can leverage the spatial U-Net as is for video generation. We achieve this by shifting two portions of the feature map channels forward and backward along the temporal dimension. The shifted features of the current frame thus receive the features from the previous and the subsequent frames, enabling motion learning without additional parameters. We show that Latent-Shift achieves comparable or better results while being significantly more efficient. Moreover, Latent-Shift can generate images despite being finetuned for T2V generation.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 37 of its 46 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

37 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  5. 2022Video Diffusion Models7 Apr

Other references · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
20 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  4. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  6. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  7. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  9. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  10. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  11. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  12. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  13. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  14. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  15. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  16. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  17. 2017Video Generation From Textcited by 153
  18. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  19. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  20. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 57; 2025: 34; 2026: 13.

137 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  5. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  6. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  7. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  8. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  9. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  10. 2023A Survey on Video Diffusion Modelscited by 203
  11. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  12. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
38 more
  1. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  2. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  3. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  4. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  5. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  6. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  7. 2023DiffusionSat: A Generative Foundation Model for Satellite Imagerycited by 130
  8. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  9. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  10. 2025VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Modelscited by 112
  11. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  12. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  13. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  14. 2023VideoLCM: Video Latent Consistency Modelcited by 90
  15. 2024Animate-X: Universal Character Image Animation with Enhanced Motion Representationcited by 86
  16. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  17. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  18. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  19. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 73
  20. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  21. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilitiescited by 58
  22. 2025Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Modelscited by 58
  23. 2024360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Modelcited by 56
  24. 20244K4DGen: Panoramic 4D Generation at 4K Resolutioncited by 54
  25. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  26. 2024On the Content Bias in Fr'echet Video Distancecited by 51
  27. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generationcited by 50
  28. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  29. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 48
  30. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
  31. 2023Breathing Life Into Sketches Using Text-to-Video Priorscited by 47
  32. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  33. 2024Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditionscited by 42
  34. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 42
  35. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
  36. 2023GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generationcited by 36
  37. 2023Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMscited by 35
  38. 2023Towards Consistent Video Editing with Text-to-Image Diffusion Modelscited by 33

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  2. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation
  3. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
  4. 2024xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
  5. 2025Mobius: Text to Seamless Looping Video Generation via Latent Shift
  6. 2024Latte: Latent Diffusion Transformer for Video Generation
  7. 2023Video Probabilistic Diffusion Models in Projected Latent Space
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models