mlsanityData to 7 Oct 2026

JVID: Joint Video-Image Diffusion for Visual-Quality and Temporal-Consistency in Video Generation

arXiv 2409.14149 ↗First posted 21 Sep 2024cs.CVcited by 0

On the map: Diffusion models → Video generation

Abstract (arXiv)

We introduce the Joint Video-Image Diffusion model (JVID), a novel approach to generating high-quality and temporally coherent videos. We achieve this by integrating two diffusion models: a Latent Image Diffusion Model (LIDM) trained on images and a Latent Video Diffusion Model (LVDM) trained on video data. Our method combines these models in the reverse diffusion process, where the LIDM enhances image quality and the LVDM ensures temporal consistency. This unique combination allows us to effectively handle the complex spatio-temporal dynamics in video generation. Our results demonstrate quantitative and qualitative improvements in producing realistic and coherent videos.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 65 of its 70 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

65 of 70 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  5. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  7. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  8. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
  9. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  11. 2022SinFusion: Training Diffusion Models on a Single Image or Video21 Nov
  12. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
6 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  3. 2022Flexible Diffusion Modeling of Long Videos23 May
  4. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  5. 2022Video Diffusion Models7 Apr
  6. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
35 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  13. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  14. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  15. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  16. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  17. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  18. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  19. 2023Zero-shot Image-to-Image Translationcited by 503
  20. 2016Generating Videos with Scene Dynamicscited by 376
  21. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  22. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  23. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  24. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  25. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  26. 2019Scaling Autoregressive Video Modelscited by 189
  27. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  28. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  29. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  30. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  31. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  32. 2020Latent Video Transformercited by 106
  33. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  34. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  35. 2019Lower Dimensional Kernels for Video Discriminatorscited by 37

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  3. 2024Video Diffusion Models: A Survey
  4. 2026Efficient Video Diffusion Models: Advancements and Challenges
  5. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
  6. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  7. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
  8. 2024Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach