# Extreme Video Compression with Pre-trained Diffusion Models

arXiv: https://arxiv.org/abs/2402.08934 · HTML: https://mlsanity.com/paper/2402.08934/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion for compression and wireless, one of 14 directions in Diffusion models.
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Placed in

- Diffusion models / [Diffusion for compression and wireless](https://mlsanity.com/diffusion/diffusion-for-compression-and-wireless/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation](https://mlsanity.com/paper/2205.09853/) (2022)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [Diffusion Probabilistic Modeling for Video Generation](https://mlsanity.com/paper/2203.09481/) (2022)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Lossy Image Compression with Conditional Diffusion Models](https://mlsanity.com/paper/2209.06950/) (2022)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Diffusion Models for Video Prediction and Infilling](https://mlsanity.com/paper/2206.07696/) (2022)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Transformation-based Adversarial Video Prediction on Large-Scale Data](https://mlsanity.com/paper/2003.04035/) (2020)
- [High-Fidelity Image Compression with Score-based Generative Models](https://mlsanity.com/paper/2305.18231/) (2023)
- [Neural Video Compression using GANs for Detail Synthesis and Propagation](https://mlsanity.com/paper/2107.12038/) (2021)
- [A Residual Diffusion Model for High Perceptual Quality Codec Augmentation](https://mlsanity.com/paper/2301.05489/) (2023)
- [Adversarial Distortion for Learned Video Compression](https://mlsanity.com/paper/2004.09508/) (2020)

## Built on by (most cited first, 6 of 6)

- [Latency-Aware Generative Semantic Communications with Pre-Trained Diffusion Models](https://mlsanity.com/paper/2403.17256/) (2024)
- [ProGVC: Progressive-based Generative Video Compression via Auto-Regressive Context Modeling](https://mlsanity.com/paper/2603.17546/) (2026)
- [Higher fidelity perceptual image and video compression with a latent conditioned residual denoising diffusion model](https://mlsanity.com/paper/2505.13152/) (2025)
- [Efficient Video Diffusion with Sparse Information Transmission for Video Compression](https://mlsanity.com/paper/2603.18501/) (2026)
- [Dense-Prior-Guided Generative Video Compression](https://mlsanity.com/paper/2604.06655/) (2026)
- [DiV-INR: Extreme Low-Bitrate Diffusion Video Compression with INR Conditioning](https://mlsanity.com/paper/2604.08329/) (2026)

## Abstract (arXiv)

Diffusion models have achieved remarkable success in generating high quality image and video data. More recently, they have also been used for image compression with high perceptual quality. In this paper, we present a novel approach to extreme video compression leveraging the predictive power of diffusion-based generative models at the decoder. The conditional diffusion model takes several neural compressed frames and generates subsequent frames. When the reconstruction quality drops below the desired level, new frames are encoded to restart prediction. The entire video is sequentially encoded to achieve a visually pleasing reconstruction, considering perceptual quality metrics such as the learned perceptual image patch similarity (LPIPS) and the Frechet video distance (FVD), at bit rates as low as 0.02 bits per pixel (bpp). Experimental results demonstrate the effectiveness of the proposed scheme compared to standard codecs such as H.264 and H.265 in the low bpp regime. The results showcase the potential of exploiting the temporal relations in video data using generative models. Code is available at: https://github.com/ElesionKyrie/Extreme-Video-Compression-With-Prediction-Using-Pre-trainded-Diffusion-Models-