# Learned Scalable Video Coding For Humans and Machines

arXiv: https://arxiv.org/abs/2307.08978 · HTML: https://mlsanity.com/paper/2307.08978/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [DVC: An End-to-end Deep Video Compression Framework](https://mlsanity.com/paper/1812.00101/) (2018)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [FVC: A New Framework towards Deep Video Compression in Feature Space](https://mlsanity.com/paper/2105.09600/) (2021)
- [Temporal Context Mining for Learned Video Compression](https://mlsanity.com/paper/2111.13850/) (2021)
- [M-LVC: Multiple Frames Prediction for Learned Video Compression](https://mlsanity.com/paper/2004.10290/) (2020)
- [Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics](https://mlsanity.com/paper/2001.03569/) (2020)
- [Content Adaptive and Error Propagation Aware Deep Video Compression](https://mlsanity.com/paper/2003.11282/) (2020)
- [Scalable Image Coding for Humans and Machines](https://mlsanity.com/paper/2107.08373/) (2021)
- [CANF-VC: Conditional Augmented Normalizing Flows for Video Compression](https://mlsanity.com/paper/2207.05315/) (2022)
- [Temporal RoI Align for Video Object Recognition](https://mlsanity.com/paper/2109.03495/) (2021)
- [Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model](https://mlsanity.com/paper/2007.04574/) (2020)
- [Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach](https://mlsanity.com/paper/2001.02915/) (2020)
- [Conditional Coding for Flexible Learned Video Compression](https://mlsanity.com/paper/2104.07930/) (2021)
- [ANFIC: Image Compression Using Augmented Normalizing Flows](https://mlsanity.com/paper/2107.08470/) (2021)
- [Latent-space scalability for multi-task collaborative intelligence](https://mlsanity.com/paper/2105.10089/) (2021)
- [Optical Flow and Mode Selection for Learning-based Video Coding](https://mlsanity.com/paper/2008.02580/) (2020)
- [Rate-Distortion Theory in Coding for Machines and its Application](https://mlsanity.com/paper/2305.17295/) (2023)
- [Scalable Video Coding for Humans and Machines](https://mlsanity.com/paper/2208.02512/) (2022)
- [Rate-Distortion in Image Coding for Machines](https://mlsanity.com/paper/2209.11694/) (2022)
- [Tencent Video Dataset (TVD): A Video Dataset for Learning-based Visual Data Compression and Analysis](https://mlsanity.com/paper/2105.05961/) (2021)
- [Base Layer Efficiency in Scalable Human-Machine Coding](https://mlsanity.com/paper/2307.02430/) (2023)

## Built on by (most cited first, 5 of 5)

- [Task-Aware Encoder Control for Deep Video Compression](https://mlsanity.com/paper/2404.04848/) (2024)
- [Scalable Image Coding for Humans and Machines Using Feature Fusion Network](https://mlsanity.com/paper/2405.09152/) (2024)
- [Explicit Residual-Based Scalable Image Coding for Humans and Machines](https://mlsanity.com/paper/2506.19297/) (2025)
- [Guided Diffusion for the Extension of Machine Vision to Human Visual Perception](https://mlsanity.com/paper/2503.17907/) (2025)
- [Scalable Neural Video Representation Compression](https://mlsanity.com/paper/2609.04273/) (2026)

## Abstract (arXiv)

Video coding has traditionally been developed to support services such as video streaming, videoconferencing, digital TV, and so on. The main intent was to enable human viewing of the encoded content. However, with the advances in deep neural networks (DNNs), encoded video is increasingly being used for automatic video analytics performed by machines. In applications such as automatic traffic monitoring, analytics such as vehicle detection, tracking and counting, would run continuously, while human viewing could be required occasionally to review potential incidents. To support such applications, a new paradigm for video coding is needed that will facilitate efficient representation and compression of video for both machine and human use in a scalable manner. In this manuscript, we introduce an end-to-end learnable video codec that supports a machine vision task in its base layer, while its enhancement layer, together with the base layer, supports input reconstruction for human viewing. The proposed system is constructed based on the concept of conditional coding to achieve better compression gains. Comprehensive experimental evaluations conducted on four standard video datasets demonstrate that our framework outperforms both state-of-the-art learned and conventional video codecs in its base layer, while maintaining comparable performance on the human vision task in its enhancement layer. Implementation of the proposed system is available at https://github.com/hadipardis/svc