# LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models

arXiv: https://arxiv.org/abs/2503.14325 · HTML: https://mlsanity.com/paper/2503.14325/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 32 of its 49 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 32)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Autoregressive Image Generation without Vector Quantization](https://mlsanity.com/paper/2406.11838/) (2024)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://mlsanity.com/paper/2112.14683/) (2021)
- [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022)
- [OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation](https://mlsanity.com/paper/2406.09399/) (2024)
- [Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks](https://mlsanity.com/paper/1709.07592/) (2017)
- [EasyAnimate: High-Performance Video Generation Framework with Hybrid Windows Attention and Reward Backpropagation](https://mlsanity.com/paper/2405.18991/) (2024)
- [CV-VAE: A Compatible Video VAE for Latent Generative Video Models](https://mlsanity.com/paper/2405.20279/) (2024)
- [WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.17459/) (2024)
- [Improved Video VAE for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.06449/) (2024)
- [VidTok: A Versatile and Open-Source Video Tokenizer](https://mlsanity.com/paper/2412.13061/) (2024)
- [Learnings from Scaling Visual Tokenizers for Reconstruction and Generation](https://mlsanity.com/paper/2501.09755/) (2025)
- [OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model](https://mlsanity.com/paper/2409.01199/) (2024)
- [LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models](https://mlsanity.com/paper/2405.14477/) (2024)
- [Large Motion Video Autoencoding with Cross-modal Video VAE](https://mlsanity.com/paper/2412.17805/) (2024)

## Built on by (most cited first, 17 of 17)

- [Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devices](https://mlsanity.com/paper/2508.09136/) (2025)
- [Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation](https://mlsanity.com/paper/2602.19161/) (2026)
- [Efficient Video Diffusion Models: Advancements and Challenges](https://mlsanity.com/paper/2604.15911/) (2026)
- [InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior](https://mlsanity.com/paper/2511.14208/) (2025)
- [Adaptive 1D Video Diffusion Autoencoder](https://mlsanity.com/paper/2602.04220/) (2026)
- [Image-to-Video Diffusion: From Foundations to Open Frontiers](https://mlsanity.com/paper/2605.17248/) (2026)
- [MobileWan: Closing the Quality Gap for Mobile Video Diffusion](https://mlsanity.com/paper/2607.06173/) (2026)
- [VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders](https://mlsanity.com/paper/2607.14088/) (2026)
- [Splannequin: Freezing Monocular Mannequin-Challenge Footage with Dual-Detection Splatting](https://mlsanity.com/paper/2512.05113/) (2025)
- [DiffInk: Glyph- and Style-Aware Latent Diffusion Transformer for Text to Online Handwriting Generation](https://mlsanity.com/paper/2509.23624/) (2025)
- [Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation](https://mlsanity.com/paper/2605.30893/) (2026)
- [Pixel-Space Diffusion Transformers](https://mlsanity.com/paper/2607.17585/) (2026)
- [Video Generation with Predictive Latents](https://mlsanity.com/paper/2605.02134/) (2026)
- [Octree-based Video Representation](https://mlsanity.com/paper/2609.33100/) (2026)
- [GRACE: Generation-aware latent compression for efficient video generation](https://mlsanity.com/paper/2610.10524/) (2026)
- [LightVAE: Towards Compact and Efficient Video Autoencoders](https://mlsanity.com/paper/2610.14726/) (2026)
- [Focus, Align, and Diffuse: Time-Series-Aware Keyframe Diffusion for Cardiac Dynamic Synthesis from Sparse Observations](https://mlsanity.com/paper/2610.19050/) (2026)

## Abstract (arXiv)

Recent advances in Latent Video Diffusion Models (LVDMs) have revolutionized video generation by leveraging Video Variational Autoencoders (Video VAEs) to compress intricate video data into a compact latent space. However, as LVDM training scales, the computational overhead of Video VAEs becomes a critical bottleneck, particularly for encoding high-resolution videos. To address this, we propose LeanVAE, a novel and ultra-efficient Video VAE framework that introduces two key innovations: (1) a lightweight architecture based on a Neighborhood-Aware Feedforward (NAF) module and non-overlapping patch operations, drastically reducing computational cost, and (2) the integration of wavelet transforms and compressed sensing techniques to enhance reconstruction quality. Extensive experiments validate LeanVAE's superiority in video reconstruction and generation, particularly in enhancing efficiency over existing Video VAEs. Our model offers up to 50x fewer FLOPs and 44x faster inference speed while maintaining competitive reconstruction quality, providing insights for scalable, efficient video generation. Our models and code are available at https://github.com/westlake-repl/LeanVAE