# Unified Coding for Both Human Perception and Generalized Machine Analytics with CLIP Supervision

arXiv: https://arxiv.org/abs/2501.04579 · HTML: https://mlsanity.com/paper/2501.04579/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining](https://mlsanity.com/paper/2204.14095/) (2022)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [Causal Contextual Prediction for Learned Image Compression](https://mlsanity.com/paper/2011.09704/) (2020)
- [Towards Image Understanding from Deep Compression without Decoding](https://mlsanity.com/paper/1803.06131/) (2018)
- [Image coding for machines: an end-to-end learned approach](https://mlsanity.com/paper/2108.09993/) (2021)
- [TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception](https://mlsanity.com/paper/2306.05085/) (2023)
- [End-to-end optimized image compression for machines, a study](https://mlsanity.com/paper/2011.06409/) (2020)
- [Towards Analysis-friendly Face Representation with Scalable Feature and Texture Compression](https://mlsanity.com/paper/2004.10043/) (2020)
- [On The Classification-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1904.08816/) (2019)
- [Visual Analysis Motivated Rate-Distortion Model for Image Coding](https://mlsanity.com/paper/2104.10315/) (2021)
- [Image Coding for Machines with Edge Information Learning Using Segment Anything](https://mlsanity.com/paper/2403.04173/) (2024)

## Built on by (most cited first, 4 of 4)

- [DT-UFC: Universal Large Model Feature Coding via Peaky-to-Balanced Distribution Transformation](https://mlsanity.com/paper/2506.16495/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters](https://mlsanity.com/paper/2604.10017/) (2026)
- [ML-CLIPSim: Multi-Layer CLIP Similarity for Machine-Oriented Image Quality](https://mlsanity.com/paper/2605.09479/) (2026)

## Abstract (arXiv)

The image compression model has long struggled with adaptability and generalization, as the decoded bitstream typically serves only human or machine needs and fails to preserve information for unseen visual tasks. Therefore, this paper innovatively introduces supervision obtained from multimodal pre-training models and incorporates adaptive multi-objective optimization tailored to support both human visual perception and machine vision simultaneously with a single bitstream, denoted as Unified and Generalized Image Coding for Machine (UG-ICM). Specifically, to get rid of the reliance between compression models with downstream task supervision, we introduce Contrastive Language-Image Pre-training (CLIP) models into the training constraint for improved generalization. Global-to-instance-wise CLIP supervision is applied to help obtain hierarchical semantics that make models more generalizable for the tasks relying on the information of different granularity. Furthermore, for supporting both human and machine visions with only a unifying bitstream, we incorporate a conditional decoding strategy that takes as conditions human or machine preferences, enabling the bitstream to be decoded into different versions for corresponding preferences. As such, our proposed UG-ICM is fully trained in a self-supervised manner, i.e., without awareness of any specific downstream models and tasks. The extensive experiments have shown that the proposed UG-ICM is capable of achieving remarkable improvements in various unseen machine analytics tasks, while simultaneously providing perceptually satisfying images.