# Adaptation and Attention for Neural Video Coding

arXiv: https://arxiv.org/abs/2112.08767 · HTML: https://mlsanity.com/paper/2112.08767/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Selective Kernel Networks](https://mlsanity.com/paper/1903.06586/) (2019)
- [Video Compression through Image Interpolation](https://mlsanity.com/paper/1804.06919/) (2018)
- [Learning for Video Compression with Hierarchical Quality and Recurrent Enhancement](https://mlsanity.com/paper/2003.01966/) (2020)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model](https://mlsanity.com/paper/2006.13560/) (2020)
- [Content Adaptive Optimization for Neural Image Compression](https://mlsanity.com/paper/1906.01223/) (2019)
- [Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model](https://mlsanity.com/paper/2007.04574/) (2020)
- [Overfitting for Fun and Profit: Instance-Adaptive Data Compression](https://mlsanity.com/paper/2101.08687/) (2021)
- [Efficient Adaptation of Neural Network Filter for Video Compression](https://mlsanity.com/paper/2007.14267/) (2020)
- [Block-optimized Variable Bit Rate Neural Image Compression](https://mlsanity.com/paper/1805.10887/) (2018)
- [Compressing Weight-updates for Image Artifacts Removal Neural Networks](https://mlsanity.com/paper/1905.04079/) (2019)
- [Lossless Image Compression Using a Multi-Scale Progressive Statistical Model](https://mlsanity.com/paper/2108.10551/) (2021)
- [A Compression Objective and a Cycle Loss for Neural Image Compression](https://mlsanity.com/paper/1905.10371/) (2019)

## Built on by (most cited first, 6 of 6)

- [Hierarchical B-frame Video Coding Using Two-Layer CANF without Motion Coding](https://mlsanity.com/paper/2304.02690/) (2023)
- [Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression](https://mlsanity.com/paper/2405.04274/) (2024)
- [Universal Deep Image Compression via Content-Adaptive Optimization with Adapters](https://mlsanity.com/paper/2211.00918/) (2022)
- [Dynamic Low-Rank Instance Adaptation for Universal Neural Image Compression](https://mlsanity.com/paper/2308.07733/) (2023)
- [Test-time adaptation for image compression with distribution regularization](https://mlsanity.com/paper/2410.12191/) (2024)
- [Few-Shot Domain Adaptation for Learned Image Compression](https://mlsanity.com/paper/2409.11111/) (2024)

## Abstract (arXiv)

Neural image coding represents now the state-of-the-art image compression approach. However, a lot of work is still to be done in the video domain. In this work, we propose an end-to-end learned video codec that introduces several architectural novelties as well as training novelties, revolving around the concepts of adaptation and attention. Our codec is organized as an intra-frame codec paired with an inter-frame codec. As one architectural novelty, we propose to train the inter-frame codec model to adapt the motion estimation process based on the resolution of the input video. A second architectural novelty is a new neural block that combines concepts from split-attention based neural networks and from DenseNets. Finally, we propose to overfit a set of decoder-side multiplicative parameters at inference time. Through ablation studies and comparisons to prior art, we show the benefits of our proposed techniques in terms of coding gains. We compare our codec to VVC/H.266 and RLVC, which represent the state-of-the-art traditional and end-to-end learned codecs, respectively, and to the top performing end-to-end learned approach in 2021 CLIC competition, E2E_T_OL. Our codec clearly outperforms E2E_T_OL, and compare favorably to VVC and RLVC in some settings.