# Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics

arXiv: https://arxiv.org/abs/2001.03569 · HTML: https://mlsanity.com/paper/2001.03569/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2020.

## Built on by (most cited first, 30 of 80)

- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [FitVid: Overfitting in Pixel-Level Video Prediction](https://mlsanity.com/paper/2106.13195/) (2021)
- [Wireless Deep Video Semantic Transmission](https://mlsanity.com/paper/2205.13129/) (2022)
- [USTC-TD: A Test Dataset and Benchmark for Image and Video Coding in 2020s](https://mlsanity.com/paper/2409.08481/) (2024)
- [Image Coding for Machines with Omnipotent Feature Learning](https://mlsanity.com/paper/2207.01932/) (2022)
- [Task-Oriented Image Transmission for Scene Classification in Unmanned Aerial Systems](https://mlsanity.com/paper/2112.10948/) (2021)
- [Towards End-to-End Image Compression and Analysis with Transformers](https://mlsanity.com/paper/2112.09300/) (2021)
- [Image Compression for Machine and Human Vision with Spatial-Frequency Adaptation](https://mlsanity.com/paper/2407.09853/) (2024)
- [Collaborative Intelligence: Challenges and Opportunities](https://mlsanity.com/paper/2102.06841/) (2021)
- [A Coding Framework and Benchmark towards Low-Bitrate Video Understanding](https://mlsanity.com/paper/2202.02813/) (2022)
- [Recent Standard Development Activities on Video Coding for Machines](https://mlsanity.com/paper/2105.12653/) (2021)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [End-to-end Compression Towards Machine Vision: Network Architecture Design and Optimization](https://mlsanity.com/paper/2107.00328/) (2021)
- [SMC++: Masked Learning of Unsupervised Video Semantic Compression](https://mlsanity.com/paper/2406.04765/) (2024)
- [Video Coding for Machine: Compact Visual Representation Compression for Intelligent Collaborative Analytics](https://mlsanity.com/paper/2110.09241/) (2021)
- [Scalable Face Image Coding via StyleGAN Prior: Towards Compression for Human-Machine Collaborative Vision](https://mlsanity.com/paper/2312.15622/) (2023)
- [Rate-Distortion-Cognition Controllable Versatile Neural Image Compression](https://mlsanity.com/paper/2407.11700/) (2024)
- [End-to-end optimized image compression for multiple machine tasks](https://mlsanity.com/paper/2103.04178/) (2021)
- [Scalable Video Coding for Humans and Machines](https://mlsanity.com/paper/2208.02512/) (2022)
- [An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation](https://mlsanity.com/paper/2201.12477/) (2022)
- [Video Coding for Machines with Feature-Based Rate-Distortion Optimization](https://mlsanity.com/paper/2203.05890/) (2022)
- [Toward Adaptive Semantic Communications: Efficient Data Transmission via Online Learned Nonlinear Transform Source-Channel Coding](https://mlsanity.com/paper/2211.04339/) (2022)
- [Semantically Video Coding: Instill Static-Dynamic Clues into Structured Bitstream for AI Tasks](https://mlsanity.com/paper/2201.10162/) (2022)
- [Feature Compression for Rate Constrained Object Detection on the Edge](https://mlsanity.com/paper/2204.07314/) (2022)
- [AccMPEG: Optimizing Video Encoding for Video Analytics](https://mlsanity.com/paper/2204.12534/) (2022)
- [Prompt-ICM: A Unified Framework towards Image Coding for Machines with Task-driven Prompts](https://mlsanity.com/paper/2305.02578/) (2023)
- [Image Coding for Machines with Object Region Learning](https://mlsanity.com/paper/2308.13984/) (2023)
- [Machine Perception-Driven Image Compression: A Layered Generative Approach](https://mlsanity.com/paper/2304.06896/) (2023)
- [Scalable Image Coding for Humans and Machines Using Feature Fusion Network](https://mlsanity.com/paper/2405.09152/) (2024)
- [Toward Scalable Image Feature Compression: A Content-Adaptive and Diffusion-Based Approach](https://mlsanity.com/paper/2410.06149/) (2024)

## Abstract (arXiv)

Video coding, which targets to compress and reconstruct the whole frame, and feature compression, which only preserves and transmits the most critical information, stand at two ends of the scale. That is, one is with compactness and efficiency to serve for machine vision, and the other is with full fidelity, bowing to human perception. The recent endeavors in imminent trends of video compression, e.g. deep learning based coding tools and end-to-end image/video coding, and MPEG-7 compact feature descriptor standards, i.e. Compact Descriptors for Visual Search and Compact Descriptors for Video Analysis, promote the sustainable and fast development in their own directions, respectively. In this paper, thanks to booming AI technology, e.g. prediction and generation models, we carry out exploration in the new area, Video Coding for Machines (VCM), arising from the emerging MPEG standardization efforts1. Towards collaborative compression and intelligent analytics, VCM attempts to bridge the gap between feature coding for machine vision and video coding for human vision. Aligning with the rising Analyze then Compress instance Digital Retina, the definition, formulation, and paradigm of VCM are given first. Meanwhile, we systematically review state-of-the-art techniques in video compression and feature compression from the unique perspective of MPEG standardization, which provides the academic and industrial evidence to realize the collaborative compression of video and feature streams in a broad range of AI applications. Finally, we come up with potential VCM solutions, and the preliminary results have demonstrated the performance and efficiency gains. Further direction is discussed as well.