# Hybrid Video Diffusion Models with 2D Triplane and 3D Wavelet Representation

arXiv: https://arxiv.org/abs/2402.13729 · HTML: https://mlsanity.com/paper/2402.13729/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 19 of 19)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [Scaling Autoregressive Video Models](https://mlsanity.com/paper/1906.02634/) (2019)
- [Video Probabilistic Diffusion Models in Projected Latent Space](https://mlsanity.com/paper/2302.07685/) (2023)
- [Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation](https://mlsanity.com/paper/2304.08477/) (2023)
- [Compositional Foundation Models for Hierarchical Planning](https://mlsanity.com/paper/2309.08587/) (2023)
- [LaMD: Latent Motion Diffusion for Image-Conditional Video Generation](https://mlsanity.com/paper/2304.11603/) (2023)

## Built on by (most cited first, 10 of 10)

- [Improved Video VAE for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.06449/) (2024)
- [Spectral Motion Alignment for Video Motion Transfer using Diffusion Models](https://mlsanity.com/paper/2403.15249/) (2024)
- [LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models](https://mlsanity.com/paper/2503.14325/) (2025)
- [MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation](https://mlsanity.com/paper/2502.12632/) (2025)
- [Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction](https://mlsanity.com/paper/2411.14762/) (2024)
- [PRISM: Privacy-Preserving Improved Stochastic Masking for Federated Generative Models](https://mlsanity.com/paper/2503.08085/) (2025)
- [Understanding Flatness in Generative Models: Its Role and Benefits](https://mlsanity.com/paper/2503.11078/) (2025)
- [TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization](https://mlsanity.com/paper/2606.17590/) (2026)
- [Factorized Video Autoencoders for Efficient Generative Modelling](https://mlsanity.com/paper/2412.04452/) (2024)
- [Label-free Motion-Conditioned Diffusion Model for Cardiac Ultrasound Synthesis](https://mlsanity.com/paper/2512.09418/) (2025)

## Abstract (arXiv)

Generating high-quality videos that synthesize desired realistic content is a challenging task due to their intricate high-dimensionality and complexity of videos. Several recent diffusion-based methods have shown comparable performance by compressing videos to a lower-dimensional latent space, using traditional video autoencoder architecture. However, such method that employ standard frame-wise 2D and 3D convolution fail to fully exploit the spatio-temporal nature of videos. To address this issue, we propose a novel hybrid video diffusion model, called HVDM, which can capture spatio-temporal dependencies more effectively. The HVDM is trained by a hybrid video autoencoder which extracts a disentangled representation of the video including: (i) a global context information captured by a 2D projected latent (ii) a local volume information captured by 3D convolutions with wavelet decomposition (iii) a frequency information for improving the video reconstruction. Based on this disentangled representation, our hybrid autoencoder provide a more comprehensive video latent enriching the generated videos with fine structures and details. Experiments on video generation benchamarks (UCF101, SkyTimelapse, and TaiChi) demonstrate that the proposed approach achieves state-of-the-art video generation quality, showing a wide range of video applications (e.g., long video generation, image-to-video, and video dynamics control).