mlsanityData to 13 Oct 2026

Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation

arXiv 2309.15818 ↗First posted 27 Sep 2023cs.CVcited by 301

On the map: Diffusion models → Video generation

Abstract (arXiv)

Significant advancements have been achieved in the realm of large-scale pre-trained text-to-video Diffusion Models (VDMs). However, previous methods either rely solely on pixel-based VDMs, which come with high computational costs, or on latent-based VDMs, which often struggle with precise text-video alignment. In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation. Our model first uses pixel-based VDMs to produce a low-resolution video of strong text-video correlation. After that, we propose a novel expert translation method that employs the latent-based VDMs to further upsample the low-resolution video to high resolution, which can also remove potential artifacts and corruptions from low-resolution videos. Compared to latent VDMs, Show-1 can produce high-quality videos of precise text-video alignment; Compared to pixel VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G vs 72G). Furthermore, our Show-1 model can be readily adapted for motion customization and video stylization applications through simple temporal attention layer finetuning. Our model achieves state-of-the-art performance on standard video generation benchmarks. Our code and model weights are publicly available at https://github.com/showlab/Show-1.

Key facts. Cited by 301 papers in the map (data through 13 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 57 of its 69 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

57 of 69 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  2. 2023ModelScope Text-to-Video Technical Report12 Aug
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  6. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  7. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  8. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  9. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  11. 2022SinFusion: Training Diffusion Models on a Single Image or Video21 Nov
  12. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
6 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  3. 2022Flexible Diffusion Modeling of Long Videos23 May
  4. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  5. 2022Video Diffusion Models7 Apr
  6. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 39

  1. 2017Attention Is All You Needcited by 60,106
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  9. 2022Generative Adversarial Networkscited by 5,614
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,141
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,605
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,921
27 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,732
  2. 2021Palette: Image-to-Image Diffusion Modelscited by 1,298
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,217
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  5. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,159
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 975
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 926
  8. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 653
  12. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  13. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  14. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  15. 2016Generating Videos with Scene Dynamicscited by 376
  16. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 301
  17. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  18. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 255
  19. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  20. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  21. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  22. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  23. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  24. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  25. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  26. 2021CCVS: Context-aware Controllable Video Synthesiscited by 69
  27. 2023MoStGAN-V: Video Generation with Temporal Motion Stylescited by 37

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 32; 2024: 126; 2025: 94; 2026: 49.

301 papers.

Sort:

By area:

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,218
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,775
  3. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,625
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 768
  5. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 571
  6. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 482
  7. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 466
  8. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 451
  9. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 357
  10. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 334
  11. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 334
  12. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 297
38 more
  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 245
  2. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 244
  3. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 241
  4. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  5. 2023A Survey on Video Diffusion Modelscited by 206
  6. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  7. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 178
  8. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  9. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  10. 2025Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Modelscited by 160
  11. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  12. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 152
  13. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  14. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  15. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 118
  16. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  17. 2025Long Context Tuning for Video Generationcited by 111
  18. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  19. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 102
  20. 2024Loong: Generating Minute-level Long Videos with Autoregressive Language Modelscited by 99
  21. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Modelscited by 94
  22. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 93
  23. 2023VideoLCM: Video Latent Consistency Modelcited by 91
  24. 2024RingID: Rethinking Tree-Ring Watermarking for Enhanced Multi-Key Identificationcited by 91
  25. 2025Mixture of Contexts for Long Video Generationcited by 91
  26. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 90
  27. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Modelscited by 86
  28. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  29. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 85
  30. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjectscited by 77
  31. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 75
  32. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  33. 2024Vlogger: Make Your Dream A Vlogcited by 70
  34. 2024CV-VAE: A Compatible Video VAE for Latent Generative Video Modelscited by 70
  35. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  36. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 69
  37. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 67
  38. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 66

50 most cited of 301 All 301 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  2. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  3. 2024xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
  4. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  5. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
  6. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search
  7. 2026Early Failure Detection and Intervention in Video Diffusion Models
  8. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation