# Unicom: Universal and Compact Representation Learning for Image Retrieval

arXiv: https://arxiv.org/abs/2304.05884 · HTML: https://mlsanity.com/paper/2304.05884/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 77 references that are in the map.
- First posted on arXiv 12 Apr 2023.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Circle Loss: A Unified Perspective of Pair Similarity Optimization](https://mlsanity.com/paper/2002.10857/) (2020)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)

## Built on by (most cited first, 30 of 36)

- [ALIP: Adaptive Language-Image Pre-training with Synthetic Caption](https://mlsanity.com/paper/2308.08428/) (2023)
- [ILIAS: Instance-Level Image retrieval At Scale](https://mlsanity.com/paper/2502.11748/) (2025)
- [RWKV-CLIP: A Robust Vision-Language Representation Learner](https://mlsanity.com/paper/2406.06973/) (2024)
- [Target before Shooting: Accurate Anomaly Detection and Localization under One Millisecond via Cascade Patch Retrieval](https://mlsanity.com/paper/2308.06748/) (2023)
- [CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination](https://mlsanity.com/paper/2408.09441/) (2024)
- [Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives](https://mlsanity.com/paper/2404.11317/) (2024)
- [Multi-label Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2407.17331/) (2024)
- [Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval](https://mlsanity.com/paper/2403.05261/) (2024)
- [Region-based Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2507.20025/) (2025)
- [OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence](https://mlsanity.com/paper/2602.08683/) (2026)
- [Data-Efficient Multimodal Fusion on a Single GPU](https://mlsanity.com/paper/2312.10144/) (2023)
- [Bringing Multimodality to Amazon Visual Search System](https://mlsanity.com/paper/2412.13364/) (2024)
- [FORB: A Flat Object Retrieval Benchmark for Universal Image Embedding](https://mlsanity.com/paper/2309.16249/) (2023)
- [ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Images](https://mlsanity.com/paper/2408.17027/) (2024)
- [From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoning](https://mlsanity.com/paper/2511.23031/) (2025)
- [Threshold-Consistent Margin Loss for Open-World Deep Metric Learning](https://mlsanity.com/paper/2307.04047/) (2023)
- [RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm](https://mlsanity.com/paper/2502.12513/) (2025)
- [Self-Supervised Representation Learning for Adversarial Attack Detection](https://mlsanity.com/paper/2407.04382/) (2024)
- [Generalizable Object Re-Identification via Visual In-Context Prompting](https://mlsanity.com/paper/2508.21222/) (2025)
- [Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval](https://mlsanity.com/paper/2509.09118/) (2025)
- [The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learning](https://mlsanity.com/paper/2602.02117/) (2026)
- [Open-World Dynamic Prompt and Continual Visual Representation Learning](https://mlsanity.com/paper/2409.05312/) (2024)
- [Multimodal Pathology Image Search Between H&E Slides and Multiplexed Immunofluorescent Images](https://mlsanity.com/paper/2306.06780/) (2023)
- [MYCROFT: Towards Effective and Efficient External Data Augmentation](https://mlsanity.com/paper/2410.08432/) (2024)
- [Enhancing Sample Utilization in Noise-Robust Deep Metric Learning With Subgroup-Based Positive-Pair Selection](https://mlsanity.com/paper/2501.11063/) (2025)
- [UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrieval](https://mlsanity.com/paper/2508.04136/) (2025)
- [Instance-Level Generation for Representation Learning](https://mlsanity.com/paper/2510.09171/) (2025)
- [ELViS: Efficient Visual Similarity from Local Descriptors that Generalizes Across Domains](https://mlsanity.com/paper/2603.28603/) (2026)
- [Illuminating Visual Identity in Universal Multimodal Embeddings](https://mlsanity.com/paper/2608.01794/) (2026)
- [DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer](https://mlsanity.com/paper/2401.12820/) (2024)

## Abstract (arXiv)

Modern image retrieval methods typically rely on fine-tuning pre-trained encoders to extract image-level descriptors. However, the most widely used models are pre-trained on ImageNet-1K with limited classes. The pre-trained feature representation is therefore not universal enough to generalize well to the diverse open-world classes. In this paper, we first cluster the large-scale LAION400M into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model. Due to the confusion of label granularity, the automatically clustered dataset inevitably contains heavy inter-class conflict. To alleviate such conflict, we randomly select partial inter-class prototypes to construct the margin-based softmax loss. To further enhance the low-dimensional feature representation, we randomly select partial feature dimensions when calculating the similarities between embeddings and class-wise prototypes. The dual random partial selections are with respect to the class dimension and the feature dimension of the prototype matrix, making the classification conflict-robust and the feature embedding compact. Our method significantly outperforms state-of-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks. The code and pre-trained models are released to facilitate future research https://github.com/deepglint/unicom.