# Boosting neural video codecs by exploiting hierarchical redundancy

arXiv: https://arxiv.org/abs/2208.04303 · HTML: https://mlsanity.com/paper/2208.04303/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 8 Aug 2022.

## Builds on (most cited first, 24 of 24)

- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018)
- [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016)
- [Implicit Neural Representations for Image Compression](https://mlsanity.com/paper/2112.04267/) (2021)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [Perceptual Learned Video Compression with Recurrent Conditional GAN](https://mlsanity.com/paper/2109.03082/) (2021)
- [Overfitting for Fun and Profit: Instance-Adaptive Data Compression](https://mlsanity.com/paper/2101.08687/) (2021)
- [Neural Video Compression using GANs for Detail Synthesis and Propagation](https://mlsanity.com/paper/2107.12038/) (2021)
- [Feedback Recurrent Autoencoder for Video Compression](https://mlsanity.com/paper/2004.04342/) (2020)
- [Extending Neural P-frame Codecs for B-frame Coding](https://mlsanity.com/paper/2104.00531/) (2021)
- [Conditional Coding for Flexible Learned Video Compression](https://mlsanity.com/paper/2104.07930/) (2021)
- [Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set](https://mlsanity.com/paper/2111.10302/) (2021)
- [Transform Network Architectures for Deep Learning based End-to-End Image/Video Coding in Subsampled Color Spaces](https://mlsanity.com/paper/2103.01760/) (2021)
- [Learned Video Compression with Residual Prediction and Loop Filter](https://mlsanity.com/paper/2108.08551/) (2021)
- [A Combined Deep Learning based End-to-End Video Coding Architecture for YUV Color Space](https://mlsanity.com/paper/2104.00807/) (2021)

## Built on by (most cited first, 5 of 5)

- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Towards Practical Real-Time Neural Video Compression](https://mlsanity.com/paper/2502.20762/) (2025)
- [MobileNVC: Real-time 1080p Neural Video Compression on a Mobile Device](https://mlsanity.com/paper/2310.01258/) (2023)
- [Low-Latency Neural Stereo Streaming](https://mlsanity.com/paper/2403.17879/) (2024)
- [ScalablePromptus: Scalable and High-Fidelity Prompt-Based Video Streaming](https://mlsanity.com/paper/2607.26106/) (2026)

## Abstract (arXiv)

In video compression, coding efficiency is improved by reusing pixels from previously decoded frames via motion and residual compensation. We define two levels of hierarchical redundancy in video frames: 1) first-order: redundancy in pixel space, i.e., similarities in pixel values across neighboring frames, which is effectively captured using motion and residual compensation, 2) second-order: redundancy in motion and residual maps due to smooth motion in natural videos. While most of the existing neural video coding literature addresses first-order redundancy, we tackle the problem of capturing second-order redundancy in neural video codecs via predictors. We introduce generic motion and residual predictors that learn to extrapolate from previously decoded data. These predictors are lightweight, and can be employed with most neural video codecs in order to improve their rate-distortion performance. Moreover, while RGB is the dominant colorspace in neural video coding literature, we introduce general modifications for neural video codecs to embrace the YUV420 colorspace and report YUV420 results. Our experiments show that using our predictors with a well-known neural video codec leads to 38% and 34% bitrate savings in RGB and YUV420 colorspaces measured on the UVG dataset.