# GRACE: Generation-aware latent compression for efficient video generation

arXiv: https://arxiv.org/abs/2610.10524 · HTML: https://mlsanity.com/paper/2610.10524/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 36 of its 41 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 36)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [LTX-Video: Realtime Video Latent Diffusion](https://mlsanity.com/paper/2501.00103/) (2024)
- [UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Models](https://mlsanity.com/paper/2302.04867/) (2023)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [Diffusion Transformers with Representation Autoencoders](https://mlsanity.com/paper/2510.11690/) (2025)
- [Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models](https://mlsanity.com/paper/2410.10733/) (2024)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model](https://mlsanity.com/paper/2502.10248/) (2025)
- [Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200k](https://mlsanity.com/paper/2503.09642/) (2025)
- [V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning](https://mlsanity.com/paper/2603.14482/) (2026)
- [T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback](https://mlsanity.com/paper/2405.18750/) (2024)
- [CV-VAE: A Compatible Video VAE for Latent Generative Video Models](https://mlsanity.com/paper/2405.20279/) (2024)
- [WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.17459/) (2024)
- [Improved Video VAE for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.06449/) (2024)
- [DC-AE 1.5: Accelerating Diffusion Model Convergence with Structured Latent Space](https://mlsanity.com/paper/2508.00413/) (2025)
- [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2411.13503/) (2024)
- [AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Data](https://mlsanity.com/paper/2402.00769/) (2024)

## Abstract (arXiv)

Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far fewer tokens. However, such autoencoders are challenging to train, since a higher compression ratio degrades reconstruction quality and recovering it requires more channels, which is known to slow the convergence of the DiT. The compressed latent also differs from the one the DiT was trained on, so the pretrained DiT must be either retrained from scratch or adapted at considerable cost. Compressing the autoencoder the DiT was trained with appears to preserve compatibility, yet optimizing it for reconstruction alone still shifts the latent away from the distribution the DiT has learned. To address this, we propose Generation-Aware Latent Compression for Efficient Video Generation (GRACE), a two-stage framework that compresses a pretrained video autoencoder while keeping it compatible with the pretrained DiT. Specifically, we keep a frozen base latent from the pretrained encoder and learn a residual latent for the information lost under stronger compression, while aligning the compressed latent with the pretrained latent in the feature space of the frozen DiT so that the autoencoder is optimized for generation. We then adapt the DiT with lightweight fine-tuning and asymmetric denoising, where the base is denoised ahead of the residual. GRACE reduces the token count of Wan2.1-I2V-14B by 8x and its latency by 11.1x at 480x832x81, while matching the generation quality of the pretrained pipeline before compression on VBench.