# Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics

arXiv: https://arxiv.org/abs/2110.09241 · HTML: https://mlsanity.com/paper/2110.09241/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 77 references that are in the map.
- First posted on arXiv 18 Oct 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2011.10043/) (2020)
- [Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things](https://mlsanity.com/paper/2011.08612/) (2020)
- [Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics](https://mlsanity.com/paper/2001.03569/) (2020)
- [Scalable Image Coding for Humans and Machines](https://mlsanity.com/paper/2107.08373/) (2021)
- [Image coding for machines: an end-to-end learned approach](https://mlsanity.com/paper/2108.09993/) (2021)
- [End-to-end Learning of Compressible Features](https://mlsanity.com/paper/2007.11797/) (2020)
- [Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion](https://mlsanity.com/paper/2009.05757/) (2020)
- [Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach](https://mlsanity.com/paper/2001.02915/) (2020)
- [End-to-end optimized image compression for machines, a study](https://mlsanity.com/paper/2011.06409/) (2020)
- [Towards Analysis-friendly Face Representation with Scalable Feature and Texture Compression](https://mlsanity.com/paper/2004.10043/) (2020)
- [DSSLIC: Deep Semantic Segmentation-based Layered Image Compression](https://mlsanity.com/paper/1806.03348/) (2018)
- [Low Bandwidth Video-Chat Compression using Deep Generative Models](https://mlsanity.com/paper/2012.00328/) (2020)
- [Visual Analysis Motivated Rate-Distortion Model for Image Coding](https://mlsanity.com/paper/2104.10315/) (2021)
- [Learned Image Coding for Machines: A Content-Adaptive Approach](https://mlsanity.com/paper/2108.09992/) (2021)
- [Multi-task learning with compressible features for Collaborative Intelligence](https://mlsanity.com/paper/1902.05179/) (2019)
- [Collaborative Intelligence: Challenges and Opportunities](https://mlsanity.com/paper/2102.06841/) (2021)
- [Pareto-Optimal Bit Allocation for Collaborative Intelligence](https://mlsanity.com/paper/2009.12430/) (2020)
- [Back-and-Forth prediction for deep tensor compression](https://mlsanity.com/paper/2002.07036/) (2020)
- [Recent Standard Development Activities on Video Coding for Machines](https://mlsanity.com/paper/2105.12653/) (2021)

## Built on by (most cited first, 11 of 11)

- [A Coding Framework and Benchmark towards Low-Bitrate Video Understanding](https://mlsanity.com/paper/2202.02813/) (2022)
- [Perceptual Video Coding for Machines via Satisfied Machine Ratio Modeling](https://mlsanity.com/paper/2211.06797/) (2022)
- [DMOFC: Discrimination Metric-Optimized Feature Compression](https://mlsanity.com/paper/2405.04044/) (2024)
- [Machine Perception-Driven Image Compression: A Layered Generative Approach](https://mlsanity.com/paper/2304.06896/) (2023)
- [ROI-Guided Point Cloud Geometry Compression Towards Human and Machine Vision](https://mlsanity.com/paper/2504.14240/) (2025)
- [Task Oriented Video Coding: A Survey](https://mlsanity.com/paper/2208.07313/) (2022)
- [Bridging the gap between image coding for machines and humans](https://mlsanity.com/paper/2401.10732/) (2024)
- [Competitive Learning for Achieving Content-specific Filters in Video Coding for Machines](https://mlsanity.com/paper/2406.12367/) (2024)
- [Differentially Encoded Observation Spaces for Perceptive Reinforcement Learning](https://mlsanity.com/paper/2310.01767/) (2023)
- [MetaMorphosis: Task-oriented Privacy Cognizant Feature Generation for Multi-task Learning](https://mlsanity.com/paper/2305.07815/) (2023)
- [A Joint Visual Compression and Perception Framework for Neuralmorphic Spiking Camera](https://mlsanity.com/paper/2503.02725/) (2025)

## Abstract (arXiv)

Video Coding for Machines (VCM) is committed to bridging to an extent separate research tracks of video/image compression and feature compression, and attempts to optimize compactness and efficiency jointly from a unified perspective of high accuracy machine vision and full fidelity human vision. In this paper, we summarize VCM methodology and philosophy based on existing academia and industrial efforts. The development of VCM follows a general rate-distortion optimization, and the categorization of key modules or techniques is established. From previous works, it is demonstrated that, although existing works attempt to reveal the nature of scalable representation in bits when dealing with machine and human vision tasks, there remains a rare study in the generality of low bit rate representation, and accordingly how to support a variety of visual analytic tasks. Therefore, we investigate a novel visual information compression for the analytics taxonomy problem to strengthen the capability of compact visual representations extracted from multiple tasks for visual analytics. A new perspective of task relationships versus compression is revisited. By keeping in mind the transferability among different machine vision tasks (e.g. high-level semantic and mid-level geometry-related), we aim to support multiple tasks jointly at low bit rates. In particular, to narrow the dimensionality gap between neural network generated features extracted from pixels and a variety of machine vision features/labels (e.g. scene class, segmentation labels), a codebook hyperprior is designed to compress the neural network-generated features. As demonstrated in our experiments, this new hyperprior model is expected to improve feature compression efficiency by estimating the signal entropy more accurately, which enables further investigation of the granularity of abstracting compact features among different tasks.