# Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set

arXiv: https://arxiv.org/abs/2111.10302 · HTML: https://mlsanity.com/paper/2111.10302/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 83 references that are in the map.
- First posted on arXiv 19 Nov 2021.

## Builds on (most cited first, 30 of 39)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation](https://mlsanity.com/paper/1712.00080/) (2017)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [COIN: COmpression with Implicit Neural representations](https://mlsanity.com/paper/2103.03123/) (2021)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Compression With Rate-Distortion Autoencoders](https://mlsanity.com/paper/1908.05717/) (2019)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Learned Video Compression](https://mlsanity.com/paper/1811.06981/) (2018)
- [Improving Inference for Neural Image Compression](https://mlsanity.com/paper/2006.04240/) (2020)
- [Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression](https://mlsanity.com/paper/2003.08935/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [ELF-VC: Efficient Learned Flexible-Rate Video Coding](https://mlsanity.com/paper/2104.14335/) (2021)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Improving Deep Video Compression by Resolution-adaptive Flow Coding](https://mlsanity.com/paper/2009.05982/) (2020)
- [Content Adaptive Optimization for Neural Image Compression](https://mlsanity.com/paper/1906.01223/) (2019)
- [Overfitting for Fun and Profit: Instance-Adaptive Data Compression](https://mlsanity.com/paper/2101.08687/) (2021)
- [Feedback Recurrent Autoencoder for Video Compression](https://mlsanity.com/paper/2004.04342/) (2020)
- [Extending Neural P-frame Codecs for B-frame Coding](https://mlsanity.com/paper/2104.00531/) (2021)
- [Learned Video Compression via Joint Spatial-Temporal Correlation Exploration](https://mlsanity.com/paper/1912.06348/) (2019)

## Built on by (most cited first, 18 of 18)

- [HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation](https://mlsanity.com/paper/2306.09818/) (2023)
- [C3: High-performance and low-complexity neural compression from a single image or video](https://mlsanity.com/paper/2312.02753/) (2023)
- [Implicit Neural Video Compression](https://mlsanity.com/paper/2112.11312/) (2021)
- [Neural Data-Dependent Transform for Learned Image Compression](https://mlsanity.com/paper/2203.04963/) (2022)
- [MobileNVC: Real-time 1080p Neural Video Compression on a Mobile Device](https://mlsanity.com/paper/2310.01258/) (2023)
- [Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression](https://mlsanity.com/paper/2405.04274/) (2024)
- [Dynamic Low-Rank Instance Adaptation for Universal Neural Image Compression](https://mlsanity.com/paper/2308.07733/) (2023)
- [Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding](https://mlsanity.com/paper/2211.04339/) (2022)
- [Boosting neural video codecs by exploiting hierarchical redundancy](https://mlsanity.com/paper/2208.04303/) (2022)
- [Ultra-lightweight Neural Video Representation Compression](https://mlsanity.com/paper/2512.04019/) (2025)
- [NU-Class Net: A Novel Approach for Video Quality Enhancement](https://mlsanity.com/paper/2401.01163/) (2024)
- [Improving The Reconstruction Quality by Overfitted Decoder Bias in Neural Image Compression](https://mlsanity.com/paper/2210.04898/) (2022)
- [Low-Latency Neural Stereo Streaming](https://mlsanity.com/paper/2403.17879/) (2024)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)
- [MCUCoder: Adaptive Bitrate Learned Video Compression for IoT Devices](https://mlsanity.com/paper/2411.19442/) (2024)
- [Content-Adaptive Motion Rate Adaption for Learned Video Compression](https://mlsanity.com/paper/2302.06293/) (2023)
- [Few-Shot Domain Adaptation for Learned Image Compression](https://mlsanity.com/paper/2409.11111/) (2024)
- [Scalable Neural Video Representation Compression](https://mlsanity.com/paper/2609.04273/) (2026)

## Abstract (arXiv)

We introduce a video compression algorithm based on instance-adaptive learning. On each video sequence to be transmitted, we finetune a pretrained compression model. The optimal parameters are transmitted to the receiver along with the latent code. By entropy-coding the parameter updates under a suitable mixture model prior, we ensure that the network parameters can be encoded efficiently. This instance-adaptive compression algorithm is agnostic about the choice of base model and has the potential to improve any neural video codec. On UVG, HEVC, and Xiph datasets, our codec improves the performance of a scale-space flow model by between 21% and 27% BD-rate savings, and that of a state-of-the-art B-frame model by 17 to 20% BD-rate savings. We also demonstrate that instance-adaptive finetuning improves the robustness to domain shift. Finally, our approach reduces the capacity requirements of compression models. We show that it enables a competitive performance even after reducing the network size by 70%.