# Video Coding for Machines with Feature-Based Rate-Distortion Optimization

arXiv: https://arxiv.org/abs/2203.05890 · HTML: https://mlsanity.com/paper/2203.05890/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 6 of 6)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics](https://mlsanity.com/paper/2001.03569/) (2020)
- [High efficiency compression for object detection](https://mlsanity.com/paper/1710.11151/) (2017)
- [Decoding-Energy-Rate-Distortion Optimization for Video Coding](https://mlsanity.com/paper/2203.01099/) (2022)

## Built on by (most cited first, 8 of 8)

- [Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling](https://mlsanity.com/paper/2211.06797/) (2022)
- [Scalable Image Coding for Humans and Machines Using Feature Fusion Network](https://mlsanity.com/paper/2405.09152/) (2024)
- [Task Oriented Video Coding: A Survey](https://mlsanity.com/paper/2208.07313/) (2022)
- [RL-RC-DoT: A Block-level RL agent for Task-Aware Video Compression](https://mlsanity.com/paper/2501.12216/) (2025)
- [Learning Frequency-Specific Quantization Scaling in VVC for Standard-Compliant Task-driven Image Coding](https://mlsanity.com/paper/2301.08533/) (2023)
- [NN-VVC: Versatile Video Coding boosted by self-supervisedly learned image coding for machines](https://mlsanity.com/paper/2401.10761/) (2024)
- [Image Coding for Machines via Feature-Preserving Rate-Distortion Optimization](https://mlsanity.com/paper/2504.02216/) (2025)
- [A Joint Visual Compression and Perception Framework for Neuralmorphic Spiking Camera](https://mlsanity.com/paper/2503.02725/) (2025)

## Abstract (arXiv)

Common state-of-the-art video codecs are optimized to deliver a low bitrate by providing a certain quality for the final human observer, which is achieved by rate-distortion optimization (RDO). But, with the steady improvement of neural networks solving computer vision tasks, more and more multimedia data is not observed by humans anymore, but directly analyzed by neural networks. In this paper, we propose a standard-compliant feature-based RDO (FRDO) that is designed to increase the coding performance, when the decoded frame is analyzed by a neural network in a video coding for machine scenario. To that extent, we replace the pixel-based distortion metrics in conventional RDO of VTM-8.0 with distortion metrics calculated in the feature space created by the first layers of a neural network. Throughout several tests with the segmentation network Mask R-CNN and single images from the Cityscapes dataset, we compare the proposed FRDO and its hybrid version HFRDO with different distortion measures in the feature space against the conventional RDO. With HFRDO, up to 5.49 % bitrate can be saved compared to the VTM-8.0 implementation in terms of Bjontegaard Delta Rate and using the weighted average precision as quality metric. Additionally, allowing the encoder to vary the quantization parameter results in coding gains for the proposed HFRDO of up 9.95 % compared to conventional VTM.