# A New Image Codec Paradigm for Human and Machine Uses arXiv: https://arxiv.org/abs/2112.10071 ยท HTML: https://mlsanity.com/paper/2112.10071/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 38 of its 61 references that are in the map. - First posted on arXiv 19 Dec 2021. ## Builds on (most cited first, 30 of 38) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016) - [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016) - [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018) - [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016) - [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018) - [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020) - [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020) - [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017) - [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018) - [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016) - [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015) - [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017) - [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018) - [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018) - [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017) - [Understanding How Image Quality Affects Deep Neural Networks](https://mlsanity.com/paper/1604.04004/) (2016) - [Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks](https://mlsanity.com/paper/1703.10114/) (2017) - [Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics](https://mlsanity.com/paper/2001.03569/) (2020) - [Learning Convolutional Networks for Content-weighted Image Compression](https://mlsanity.com/paper/1703.10553/) (2017) - [Scalable Image Coding for Humans and Machines](https://mlsanity.com/paper/2107.08373/) (2021) - [Towards Image Understanding from Deep Compression without Decoding](https://mlsanity.com/paper/1803.06131/) (2018) - [Deep Convolutional AutoEncoder-based Lossy Image Compression](https://mlsanity.com/paper/1804.09535/) (2018) - [Generative Compression](https://mlsanity.com/paper/1703.01467/) (2017) - [Towards Coding for Human and Machine Vision: A Scalable Image Coding Approach](https://mlsanity.com/paper/2001.02915/) (2020) - [DSSLIC: Deep Semantic Segmentation-based Layered Image Compression](https://mlsanity.com/paper/1806.03348/) (2018) - [Efficient Nonlinear Transforms for Lossy Image Compression](https://mlsanity.com/paper/1802.00847/) (2018) ## Built on by (most cited first, 3 of 3) - [Minimum Noticeable Difference based Adversarial Privacy Preserving Image Generation](https://mlsanity.com/paper/2206.08638/) (2022) - [Customizable ROI-Based Deep Image Compression](https://mlsanity.com/paper/2507.00373/) (2025) - [NN-VVC: Versatile Video Coding boosted by self-supervisedly learned image coding for machines](https://mlsanity.com/paper/2401.10761/) (2024) ## Abstract (arXiv) With the AI of Things (AIoT) development, a huge amount of visual data, e.g., images and videos, are produced in our daily work and life. These visual data are not only used for human viewing or understanding but also for machine analysis or decision-making, e.g., intelligent surveillance, automated vehicles, and many other smart city applications. To this end, a new image codec paradigm for both human and machine uses is proposed in this work. Firstly, the high-level instance segmentation map and the low-level signal features are extracted with neural networks. Then, the instance segmentation map is further represented as a profile with the proposed 16-bit gray-scale representation. After that, both 16-bit gray-scale profile and signal features are encoded with a lossless codec. Meanwhile, an image predictor is designed and trained to achieve the general-quality image reconstruction with the 16-bit gray-scale profile and signal features. Finally, the residual map between the original image and the predicted one is compressed with a lossy codec, used for high-quality image reconstruction. With such designs, on the one hand, we can achieve scalable image compression to meet the requirements of different human consumption; on the other hand, we can directly achieve several machine vision tasks at the decoder side with the decoded 16-bit gray-scale profile, e.g., object classification, detection, and segmentation. Experimental results show that the proposed codec achieves comparable results as most learning-based codecs and outperforms the traditional codecs (e.g., BPG and JPEG2000) in terms of PSNR and MS-SSIM for image reconstruction. At the same time, it outperforms the existing codecs in terms of the mAP for object detection and segmentation.