# LightVAE: Towards Compact and Efficient Video Autoencoders

arXiv: https://arxiv.org/abs/2610.14726 · HTML: https://mlsanity.com/paper/2610.14726/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 29 of 29)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [One-step Diffusion with Distribution Matching Distillation](https://mlsanity.com/paper/2311.18828/) (2023)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://mlsanity.com/paper/2411.19108/) (2024)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/1905.00737/) (2019)
- [CrossKD: Cross-Head Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2306.11369/) (2023)
- [TinyFusion: Diffusion Transformers Learned Shallow](https://mlsanity.com/paper/2412.01199/) (2024)
- [TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times](https://mlsanity.com/paper/2512.16093/) (2025)
- [Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals](https://mlsanity.com/paper/2510.27684/) (2025)
- [Network Decoupling: From Regular to Depthwise Separable Convolutions](https://mlsanity.com/paper/1808.05517/) (2018)
- [LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models](https://mlsanity.com/paper/2503.14325/) (2025)
- [Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devices](https://mlsanity.com/paper/2508.09136/) (2025)
- [Gradient-Guided Knowledge Distillation for Object Detectors](https://mlsanity.com/paper/2303.04240/) (2023)
- [Flash-VAED: Plug-and-Play VAE Decoders for Efficient Video Generation](https://mlsanity.com/paper/2602.19161/) (2026)

## Abstract (arXiv)

Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive. Existing methods use stage-specific lightweight operators, yet either constrain spatial and temporal computation jointly or remove temporal computation altogether, without distinguishing their compression sensitivities in pretrained Conv3D. To address these limitations, we present LightVAE, a compression framework that exploits redundancy in temporal modeling and network depth. LightVAE exactly reparameterizes eligible causal Conv3D layers into an aggregate spatial response and a difference-driven temporal correction. Based on their distinct compression sensitivities, temporal low-rank compression (TLR) keeps the aggregate spatial kernel dense and applies layer-specific low-rank approximation only to the temporal correction. An algebraically equivalent folded implementation converts the arithmetic savings into practical acceleration. LightVAE further uses width-preserving block pruning to reduce depth and probe-guided feature alignment, guided by feedback from a frozen teacher suffix, to recover quality after compression. The framework also extends to video encoders. Across three video VAE backbones and two GPU platforms, LightVAE consistently accelerates decoding. For the Wan VAE family, it achieves up to 2.24× and 1.11× the decoding throughput of state-of-the-art efficient VAE decoders on NVIDIA H100 and RTX~5090, respectively, with higher DAVIS PSNR and lower LPIPS. On MiniMax-H3, it achieves up to 2.97× the original decoding throughput while retaining 99.51% of the original model's overall VBench-2.0 score. Our code is available at https://github.com/ModelTC/LightVAE.