# CLIM: Contrastive Language-Image Mosaic for Region Representation

arXiv: https://arxiv.org/abs/2312.11376 · HTML: https://mlsanity.com/paper/2312.11376/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Open-Vocabulary DETR with Conditional Matching](https://mlsanity.com/paper/2203.11876/) (2022)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [Towards Open Vocabulary Learning: A Survey](https://mlsanity.com/paper/2306.15880/) (2023)
- [Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection](https://mlsanity.com/paper/2207.03482/) (2022)
- [Aligning Bag of Regions for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2302.13996/) (2023)
- [Learning Object-Language Alignments for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2211.14843/) (2022)
- [C-MIL: Continuation Multiple Instance Learning for Weakly Supervised Object Detection](https://mlsanity.com/paper/1904.05647/) (2019)
- [Open Vocabulary Object Detection with Pseudo Bounding-Box Labels](https://mlsanity.com/paper/2111.09452/) (2021)
- [Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2303.05892/) (2023)

## Built on by (most cited first, 21 of 21)

- [DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2505.04410/) (2025)
- [Open Vocabulary Monocular 3D Object Detection](https://mlsanity.com/paper/2411.16833/) (2024)
- [OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision](https://mlsanity.com/paper/2405.17913/) (2024)
- [Refining CLIP's Spatial Awareness: A Visual-Centric Perspective](https://mlsanity.com/paper/2504.02328/) (2025)
- [Region-based Cluster Discrimination for Visual Representation Learning](https://mlsanity.com/paper/2507.20025/) (2025)
- [Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2508.11256/) (2025)
- [OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection](https://mlsanity.com/paper/2411.17761/) (2024)
- [FlexCap: Describe Anything in Images in Controllable Detail](https://mlsanity.com/paper/2403.12026/) (2024)
- [Unconstrained Open Vocabulary Image Classification: Zero-Shot Transfer from Text to Image via CLIP Inversion](https://mlsanity.com/paper/2407.11211/) (2024)
- [SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detection](https://mlsanity.com/paper/2410.05650/) (2024)
- [C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detection](https://mlsanity.com/paper/2509.23316/) (2025)
- [Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective](https://mlsanity.com/paper/2511.16170/) (2025)
- [RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2405.19854/) (2024)
- [Boosting Open-Vocabulary Object Detection by Handling Background Samples](https://mlsanity.com/paper/2410.08645/) (2024)
- [FOR: Finetuning for Object Level Open Vocabulary Image Retrieval](https://mlsanity.com/paper/2412.18806/) (2024)
- [ATAS: Any-to-Any Self-Distillation for Enhanced Open-Vocabulary Dense Prediction](https://mlsanity.com/paper/2506.08678/) (2025)
- [Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2501.16981/) (2025)
- [Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2604.07723/) (2026)
- [MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Grounding](https://mlsanity.com/paper/2605.03398/) (2026)
- [SynCLIP: Synonym-Coherent Language-Image Pretraining for Robust Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2607.11008/) (2026)
- [Fine-grained CLIP fine-tuning with self-annotated region alignment](https://mlsanity.com/paper/2607.13661/) (2026)

## Abstract (arXiv)

Detecting objects accurately from a large or open vocabulary necessitates the vision-language alignment on region representations. However, learning such a region-text alignment by obtaining high-quality box annotations with text labels or descriptions is expensive and infeasible. In contrast, collecting image-text pairs is simpler but lacks precise object location information to associate regions with texts. In this paper, we propose a novel approach called Contrastive Language-Image Mosaic (CLIM), which leverages large-scale image-text pairs effectively for aligning region and text representations. CLIM combines multiple images into a mosaicked image and treats each image as a `pseudo region'. The feature of each pseudo region is extracted and trained to be similar to the corresponding text embedding while dissimilar from others by a contrastive loss, enabling the model to learn the region-text alignment without costly box annotations. As a generally applicable approach, CLIM consistently improves different open-vocabulary object detection methods that use caption supervision. Furthermore, CLIM can effectively enhance the region representation of vision-language models, thus providing stronger backbones for open-vocabulary object detectors. Our experimental results demonstrate that CLIM improves different baseline open-vocabulary object detectors by a large margin on both OV-COCO and OV-LVIS benchmarks. The code is available at https://github.com/wusize/CLIM.