# Neural Video Compression using GANs for Detail Synthesis and Propagation

arXiv: https://arxiv.org/abs/2107.12038 · HTML: https://mlsanity.com/paper/2107.12038/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 27 of 27)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Deep Generative Models for Distribution-Preserving Lossy Compression](https://mlsanity.com/paper/1805.11057/) (2018)
- [Conditional Entropy Coding for Efficient Video Compression](https://mlsanity.com/paper/2008.09180/) (2020)
- [Generative Compression](https://mlsanity.com/paper/1703.01467/) (2017)
- [Feedback Recurrent Autoencoder for Video Compression](https://mlsanity.com/paper/2004.04342/) (2020)
- [A coding theorem for the rate-distortion-perception function](https://mlsanity.com/paper/2104.13662/) (2021)
- [Hierarchical Autoregressive Modeling for Neural Video Compression](https://mlsanity.com/paper/2010.10258/) (2020)
- [On the advantages of stochastic encoders](https://mlsanity.com/paper/2102.09270/) (2021)
- [Neural Video Compression using Spatio-Temporal Priors](https://mlsanity.com/paper/1902.07383/) (2019)
- [Lossy Compression with Distortion Constrained Optimization](https://mlsanity.com/paper/2005.04064/) (2020)

## Built on by (most cited first, 30 of 36)

- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models](https://mlsanity.com/paper/2301.11189/) (2023)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [PADL: Language-Directed Physics-Based Character Control](https://mlsanity.com/paper/2301.13868/) (2023)
- [Interactive Face Video Coding: A Generative Compression Framework](https://mlsanity.com/paper/2302.09919/) (2023)
- [Learning Cross-Scale Weighted Prediction for Efficient Neural Video Compression](https://mlsanity.com/paper/2112.13309/) (2021)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [Single-step Diffusion-based Video Coding with Semantic-Temporal Guidance](https://mlsanity.com/paper/2512.07480/) (2025)
- [Generative Latent Video Compression](https://mlsanity.com/paper/2510.09987/) (2025)
- [On the Choice of Perception Loss Function for Learned Video Compression](https://mlsanity.com/paper/2305.19301/) (2023)
- [M3-CVC: Controllable Video Compression with Multimodal Generative Models](https://mlsanity.com/paper/2411.15798/) (2024)
- [High Visual-Fidelity Learned Video Compression](https://mlsanity.com/paper/2310.04679/) (2023)
- [Rate-Distortion-Perception Tradeoff Based on the Conditional-Distribution Perception Measure](https://mlsanity.com/paper/2401.12207/) (2024)
- [Extreme Video Compression with Pre-trained Diffusion Models](https://mlsanity.com/paper/2402.08934/) (2024)
- [Task-Aware Encoder Control for Deep Video Compression](https://mlsanity.com/paper/2404.04848/) (2024)
- [Boosting neural video codecs by exploiting hierarchical redundancy](https://mlsanity.com/paper/2208.04303/) (2022)
- [GVCC: Zero-Shot Video Compression via Codebook-Driven Stochastic Rectified Flow](https://mlsanity.com/paper/2603.26571/) (2026)
- [ProGVC: Progressive-based Generative Video Compression via Auto-Regressive Context Modeling](https://mlsanity.com/paper/2603.17546/) (2026)
- [ZeroGVC: Zero-Shot Generative Video Compression with Autoregressive Diffusion Priors](https://mlsanity.com/paper/2606.22371/) (2026)
- [Learnt Deep Hyperparameter selection in Adversarial Training for compressed video enhancement with perceptual critic](https://mlsanity.com/paper/2302.14516/) (2023)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)
- [Higher fidelity perceptual image and video compression with a latent conditioned residual denoising diffusion model](https://mlsanity.com/paper/2505.13152/) (2025)
- [Efficient Video Diffusion with Sparse Information Transmission for Video Compression](https://mlsanity.com/paper/2603.18501/) (2026)
- [Dense-Prior-Guided Generative Video Compression](https://mlsanity.com/paper/2604.06655/) (2026)
- [MLVC: Multi-platform Learned Video Codec for Real-World Deployment](https://mlsanity.com/paper/2606.28027/) (2026)
- [Generative Video Compression with Adaptive Score Distillation](https://mlsanity.com/paper/2607.22772/) (2026)
- [Context-Aware Neural Video Compression on Solar Dynamics Observatory](https://mlsanity.com/paper/2309.10784/) (2023)
- [Blurry Video Compression: A Trade-off between Visual Enhancement and Data Compression](https://mlsanity.com/paper/2311.04430/) (2023)
- [Neural-based Video Compression on Solar Dynamics Observatory Images](https://mlsanity.com/paper/2407.15730/) (2024)
- [On Self-Adaptive Perception Loss Function for Sequential Lossy Compression](https://mlsanity.com/paper/2502.10628/) (2025)

## Abstract (arXiv)

We present the first neural video compression method based on generative adversarial networks (GANs). Our approach significantly outperforms previous neural and non-neural video compression methods in a user study, setting a new state-of-the-art in visual quality for neural methods. We show that the GAN loss is crucial to obtain this high visual quality. Two components make the GAN loss effective: we i) synthesize detail by conditioning the generator on a latent extracted from the warped previous reconstruction to then ii) propagate this detail with high-quality flow. We find that user studies are required to compare methods, i.e., none of our quantitative metrics were able to predict all studies. We present the network design choices in detail, and ablate them with user studies.