# SMIC: Semantic Multi-Item Compression based on CLIP dictionary

arXiv: https://arxiv.org/abs/2412.05035 · HTML: https://mlsanity.com/paper/2412.05035/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 31 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [Text + Sketch: Image Compression at Ultra Low Rates](https://mlsanity.com/paper/2307.01944/) (2023)
- [HistoGAN: Controlling Colors of GAN-Generated and Real Images via Color Histograms](https://mlsanity.com/paper/2011.11731/) (2020)
- [Image coding for machines: an end-to-end learned approach](https://mlsanity.com/paper/2108.09993/) (2021)
- [Semantic Compression With Large Language Models](https://mlsanity.com/paper/2304.12512/) (2023)
- [Learned Lossless Compression for JPEG via Frequency-Domain Prediction](https://mlsanity.com/paper/2303.02666/) (2023)

## Abstract (arXiv)

Semantic compression, a compression scheme where the distortion metric, typically MSE, is replaced with semantic fidelity metrics, tends to become more and more popular. Most recent semantic compression schemes rely on the foundation model CLIP. In this work, we extend such a scheme to image collection compression, where inter-item redundancy is taken into account during the coding phase. For that purpose, we first show that CLIP's latent space allows for easy semantic additions and subtractions. From this property, we define a dictionary-based multi-item codec that outperforms state-of-the-art generative codec in terms of compression rate, around 10-5 BPP per image, while not sacrificing semantic fidelity. We also show that the learned dictionary is of a semantic nature and works as a semantic projector for the semantic content of images.