# Accelerating Learned Video Compression via Low-Resolution Representation Learning

arXiv: https://arxiv.org/abs/2407.16418 · HTML: https://mlsanity.com/paper/2407.16418/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Optical Flow Estimation using a Spatial Pyramid Network](https://mlsanity.com/paper/1611.00850/) (2016)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [Optimizing the Latent Space of Generative Networks](https://mlsanity.com/paper/1707.05776/) (2017)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [CANF-VC: Conditional Augmented Normalizing Flows for Video Compression](https://mlsanity.com/paper/2207.05315/) (2022)
- [Neural Data-Dependent Transform for Learned Image Compression](https://mlsanity.com/paper/2203.04963/) (2022)
- [AlphaVC: High-Performance and Efficient Learned Video Compression](https://mlsanity.com/paper/2207.14678/) (2022)
- [MobileCodec: Neural Inter-frame Video Compression on Mobile Devices](https://mlsanity.com/paper/2207.08338/) (2022)
- [Advancing Learned Video Compression with In-loop Frame Prediction](https://mlsanity.com/paper/2211.07004/) (2022)
- [MMVC: Learned Multi-Mode Video Compression with Block-based Prediction Mode Selection and Density-Adaptive Entropy Coding](https://mlsanity.com/paper/2304.02273/) (2023)
- [Towards Real-Time Neural Video Codec for Cross-Platform Application Using Calibration Information](https://mlsanity.com/paper/2309.11276/) (2023)
- [Spatiotemporal Entropy Model is All You Need for Learned Video Compression](https://mlsanity.com/paper/2104.06083/) (2021)
- [Learned Video Compression with Residual Prediction and Loop Filter](https://mlsanity.com/paper/2108.08551/) (2021)

## Abstract (arXiv)

In recent years, the field of learned video compression has witnessed rapid advancement, exemplified by the latest neural video codecs DCVC-DC that has outperformed the upcoming next-generation codec ECM in terms of compression ratio. Despite this, learned video compression frameworks often exhibit low encoding and decoding speeds primarily due to their increased computational complexity and unnecessary high-resolution spatial operations, which hugely hinder their applications in reality. In this work, we introduce an efficiency-optimized framework for learned video compression that focuses on low-resolution representation learning, aiming to significantly enhance the encoding and decoding speeds. Firstly, we diminish the computational load by reducing the resolution of inter-frame propagated features obtained from reused features of decoded frames, including I-frames. We implement a joint training strategy for both the I-frame and P-frame models, further improving the compression ratio. Secondly, our approach efficiently leverages multi-frame priors for parameter prediction, minimizing computation at the decoding end. Thirdly, we revisit the application of the Online Encoder Update (OEU) strategy for high-resolution sequences, achieving notable improvements in compression ratio without compromising decoding efficiency. Our efficiency-optimized framework has significantly improved the balance between compression ratio and speed for learned video compression. In comparison to traditional codecs, our method achieves performance levels on par with the low-decay P configuration of the H.266 reference software VTM. Furthermore, when contrasted with DCVC-HEM, our approach delivers a comparable compression ratio while boosting encoding and decoding speeds by a factor of 3 and 7, respectively. On RTX 2080Ti, our method can decode each 1080p frame under 100ms.