# NN-VVC: Versatile Video Coding boosted by self-supervisedly learned image coding for machines

arXiv: https://arxiv.org/abs/2401.10761 · HTML: https://mlsanity.com/paper/2401.10761/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 19 Jan 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [VCT: A Video Compression Transformer](https://mlsanity.com/paper/2206.07307/) (2022)
- [Scalable Image Coding for Humans and Machines](https://mlsanity.com/paper/2107.08373/) (2021)
- [BVI-DVC: A Training Database for Deep Video Compression](https://mlsanity.com/paper/2003.13552/) (2020)
- [Image coding for machines: an end-to-end learned approach](https://mlsanity.com/paper/2108.09993/) (2021)
- [Learned Image Coding for Machines: A Content-Adaptive Approach](https://mlsanity.com/paper/2108.09992/) (2021)
- [ANFIC: Image Compression Using Augmented Normalizing Flows](https://mlsanity.com/paper/2107.08470/) (2021)
- [Efficient Adaptation of Neural Network Filter for Video Compression](https://mlsanity.com/paper/2007.14267/) (2020)
- [Scalable Video Coding for Humans and Machines](https://mlsanity.com/paper/2208.02512/) (2022)
- [Video Coding for Machines with Feature-Based Rate-Distortion Optimization](https://mlsanity.com/paper/2203.05890/) (2022)
- [Affine Transformation-Based Deep Frame Prediction](https://mlsanity.com/paper/2009.05666/) (2020)
- [AIVC: Artificial Intelligence based Video Codec](https://mlsanity.com/paper/2202.04365/) (2022)
- [End-to-End Learning for Video Frame Compression with Self-Attention](https://mlsanity.com/paper/2004.09226/) (2020)
- [An Efficient QP Variable Convolutional Neural Network Based In-loop Filter for Intra Coding](https://mlsanity.com/paper/2012.15003/) (2020)
- [A QP-adaptive Mechanism for CNN-based Filter in Video Coding](https://mlsanity.com/paper/2010.13059/) (2020)
- [A New Image Codec Paradigm for Human and Machine Uses](https://mlsanity.com/paper/2112.10071/) (2021)
- [Model Selection CNN-based VVC QualityEnhancement](https://mlsanity.com/paper/2105.03338/) (2021)
- [Leveraging progressive model and overfitting for efficient learned image compression](https://mlsanity.com/paper/2210.04112/) (2022)
- [Learning Frequency-Specific Quantization Scaling in VVC for Standard-Compliant Task-driven Image Coding](https://mlsanity.com/paper/2301.08533/) (2023)

## Built on by (most cited first, 1 of 1)

- [Delta-ICM: Entropy Modeling with Delta Function for Learned Image Compression](https://mlsanity.com/paper/2410.07669/) (2024)

## Abstract (arXiv)

The recent progress in artificial intelligence has led to an ever-increasing usage of images and videos by machine analysis algorithms, mainly neural networks. Nonetheless, compression, storage and transmission of media have traditionally been designed considering human beings as the viewers of the content. Recent research on image and video coding for machine analysis has progressed mainly in two almost orthogonal directions. The first is represented by end-to-end (E2E) learned codecs which, while offering high performance on image coding, are not yet on par with state-of-the-art conventional video codecs and lack interoperability. The second direction considers using the Versatile Video Coding (VVC) standard or any other conventional video codec (CVC) together with pre- and post-processing operations targeting machine analysis. While the CVC-based methods benefit from interoperability and broad hardware and software support, the machine task performance is often lower than the desired level, particularly in low bitrates. This paper proposes a hybrid codec for machines called NN-VVC, which combines the advantages of an E2E-learned image codec and a CVC to achieve high performance in both image and video coding for machines. Our experiments show that the proposed system achieved up to -43.20% and -26.8% Bjontegaard Delta rate reduction over VVC for image and video data, respectively, when evaluated on multiple different datasets and machine vision tasks. To the best of our knowledge, this is the first research paper showing a hybrid video codec that outperforms VVC on multiple datasets and multiple machine vision tasks.