# Task-Aware Encoder Control for Deep Video Compression

arXiv: https://arxiv.org/abs/2404.04848 · HTML: https://mlsanity.com/paper/2404.04848/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 49 references that are in the map.
- First posted on arXiv 7 Apr 2024.

## Builds on (most cited first, 30 of 33)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Neural Video Compression with Diverse Contexts](https://mlsanity.com/paper/2302.14402/) (2023)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [Scalable Image Coding for Humans and Machines](https://mlsanity.com/paper/2107.08373/) (2021)
- [Coarse-to-fine Deep Video Coding with Hyperprior-guided Mode Prediction](https://mlsanity.com/paper/2206.07460/) (2022)
- [Towards Image Understanding from Deep Compression without Decoding](https://mlsanity.com/paper/1803.06131/) (2018)
- [Lossy Compression for Lossless Prediction](https://mlsanity.com/paper/2106.10800/) (2021)
- [Perceptual Learned Video Compression with Recurrent Conditional GAN](https://mlsanity.com/paper/2109.03082/) (2021)
- [DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition](https://mlsanity.com/paper/1901.03460/) (2019)
- [Neural Video Compression using GANs for Detail Synthesis and Propagation](https://mlsanity.com/paper/2107.12038/) (2021)
- [Fast Object Detection in Compressed Video](https://mlsanity.com/paper/1811.11057/) (2018)
- [Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach](https://mlsanity.com/paper/2001.02915/) (2020)
- [Image Coding for Machines with Omnipotent Feature Learning](https://mlsanity.com/paper/2207.01932/) (2022)
- [Towards Analysis-friendly Face Representation with Scalable Feature and Texture Compression](https://mlsanity.com/paper/2004.10043/) (2020)
- [DSSLIC: Deep Semantic Segmentation-based Layered Image Compression](https://mlsanity.com/paper/1806.03348/) (2018)
- [Task-driven Semantic Coding via Reinforcement Learning](https://mlsanity.com/paper/2106.03511/) (2021)
- [Compact Descriptors for Video Analysis: the Emerging MPEG Standard](https://mlsanity.com/paper/1704.08141/) (2017)
- [Self-Conditioned Probabilistic Learning of Video Rescaling](https://mlsanity.com/paper/2107.11639/) (2021)
- [Preprocessing Enhanced Image Compression for Machine Vision](https://mlsanity.com/paper/2206.05650/) (2022)
- [YOLOV: Making Still Image Object Detectors Great at Video Object Detection](https://mlsanity.com/paper/2208.09686/) (2022)
- [A Coding Framework and Benchmark towards Low-Bitrate Video Understanding](https://mlsanity.com/paper/2202.02813/) (2022)
- [End-to-End Compressed Video Representation Learning for Generic Event Boundary Detection](https://mlsanity.com/paper/2203.15336/) (2022)
- [Boosting Neural Image Compression for Machines Using Latent Space Masking](https://mlsanity.com/paper/2112.08168/) (2021)

## Built on by (most cited first, 6 of 6)

- [SMC++: Masked Learning of Unsupervised Video Semantic Compression](https://mlsanity.com/paper/2406.04765/) (2024)
- [Plug-and-Play Versatile Compressed Video Enhancement](https://mlsanity.com/paper/2504.15380/) (2025)
- [RL-RC-DoT: A Block-level RL agent for Task-Aware Video Compression](https://mlsanity.com/paper/2501.12216/) (2025)
- [Motion Free B-frame Coding for Neural Video Compression](https://mlsanity.com/paper/2411.17160/) (2024)
- [Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion](https://mlsanity.com/paper/2605.02849/) (2026)
- [ML-CLIPSim: Multi-Layer CLIP Similarity for Machine-Oriented Image Quality](https://mlsanity.com/paper/2605.09479/) (2026)

## Abstract (arXiv)

Prior research on deep video compression (DVC) for machine tasks typically necessitates training a unique codec for each specific task, mandating a dedicated decoder per task. In contrast, traditional video codecs employ a flexible encoder controller, enabling the adaptation of a single codec to different tasks through mechanisms like mode prediction. Drawing inspiration from this, we introduce an innovative encoder controller for deep video compression for machines. This controller features a mode prediction and a Group of Pictures (GoP) selection module. Our approach centralizes control at the encoding stage, allowing for adaptable encoder adjustments across different tasks, such as detection and tracking, while maintaining compatibility with a standard pre-trained DVC decoder. Empirical evidence demonstrates that our method is applicable across multiple tasks with various existing pre-trained DVCs. Moreover, extensive experiments demonstrate that our method outperforms previous DVC by about 25% bitrate for different tasks, with only one pre-trained decoder.