# RL-RC-DoT: A Block-level RL agent for Task-Aware Video Compression

arXiv: https://arxiv.org/abs/2501.12216 · HTML: https://mlsanity.com/paper/2501.12216/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Eigen-CAM: Class Activation Map using Principal Components](https://mlsanity.com/paper/2008.00299/) (2020)
- [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021)
- [MuZero with Self-competition for Rate Control in VP9 Video Compression](https://mlsanity.com/paper/2202.06626/) (2022)
- [Task-driven Semantic Coding via Reinforcement Learning](https://mlsanity.com/paper/2106.03511/) (2021)
- [Video Coding for Machines with Feature-Based Rate-Distortion Optimization](https://mlsanity.com/paper/2203.05890/) (2022)
- [Reinforced Bit Allocation under Task-Driven Semantic Distortion Metrics](https://mlsanity.com/paper/1910.07392/) (2019)
- [Task-aware Distributed Source Coding under Dynamic Bandwidth](https://mlsanity.com/paper/2305.15523/) (2023)
- [Task-Aware Encoder Control for Deep Video Compression](https://mlsanity.com/paper/2404.04848/) (2024)
- [A Dual-Critic Reinforcement Learning Framework for Frame-level Bit Allocation in HEVC/H.265](https://mlsanity.com/paper/2104.01735/) (2021)
- [Hierarchical Reinforcement Learning Based Video Semantic Coding for Segmentation](https://mlsanity.com/paper/2208.11529/) (2022)
- [The Need for Medically Aware Video Compression in Gastroenterology](https://mlsanity.com/paper/2211.01472/) (2022)
- [Neural Rate Control for Video Encoding using Imitation Learning](https://mlsanity.com/paper/2012.05339/) (2020)
- [Competitive Learning for Achieving Content-specific Filters in Video Coding for Machines](https://mlsanity.com/paper/2406.12367/) (2024)
- [SiEVE: Semantically Encoded Video Analytics on Edge and Cloud](https://mlsanity.com/paper/2006.01318/) (2020)
- [Neural Frank-Wolfe Policy Optimization for Region-of-Interest Intra-Frame Coding with HEVC/H.265](https://mlsanity.com/paper/2209.13210/) (2022)
- [On Annotation-free Optimization of Video Coding for Machines](https://mlsanity.com/paper/2406.07938/) (2024)

## Built on by (most cited first, 2 of 2)

- [Revisiting MLLM Token Technology through the Lens of Classical Visual Coding](https://mlsanity.com/paper/2508.13460/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)

## Abstract (arXiv)

Video encoders optimize compression for human perception by minimizing reconstruction error under bit-rate constraints. In many modern applications such as autonomous driving, an overwhelming majority of videos serve as input for AI systems performing tasks like object recognition or segmentation, rather than being watched by humans. It is therefore useful to optimize the encoder for a downstream task instead of for perceptual image quality. However, a major challenge is how to combine such downstream optimization with existing standard video encoders, which are highly efficient and popular. Here, we address this challenge by controlling the Quantization Parameters (QPs) at the macro-block level to optimize the downstream task. This granular control allows us to prioritize encoding for task-relevant regions within each frame. We formulate this optimization problem as a Reinforcement Learning (RL) task, where the agent learns to balance long-term implications of choosing QPs on both task performance and bit-rate constraints. Notably, our policy does not require the downstream task as an input during inference, making it suitable for streaming applications and edge devices such as vehicles. We demonstrate significant improvements in two tasks, car detection, and ROI (saliency) encoding. Our approach improves task performance for a given bit rate compared to traditional task agnostic encoding methods, paving the way for more efficient task-aware video compression.