# Contrastive Localized Language-Image Pre-Training

arXiv: https://arxiv.org/abs/2410.02746 · HTML: https://mlsanity.com/paper/2410.02746/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 45 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Data Filtering Networks](https://mlsanity.com/paper/2309.17425/) (2023)
- [MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training](https://mlsanity.com/paper/2403.09611/) (2024)
- [PixelLM: Pixel Reasoning with Large Multimodal Model](https://mlsanity.com/paper/2312.02228/) (2023)
- [Improving CLIP Training with Language Rewrites](https://mlsanity.com/paper/2305.20088/) (2023)
- [DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection](https://mlsanity.com/paper/2209.09407/) (2022)

## Built on by (most cited first, 30 of 33)

- [MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs](https://mlsanity.com/paper/2503.13111/) (2025)
- [Scaling Vision Pre-Training to 4K Resolution](https://mlsanity.com/paper/2503.19903/) (2025)
- [Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought](https://mlsanity.com/paper/2505.23766/) (2025)
- [GOAL: Global-local Object Alignment Learning](https://mlsanity.com/paper/2503.17782/) (2025)
- [Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?](https://mlsanity.com/paper/2410.19546/) (2024)
- [OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning](https://mlsanity.com/paper/2505.04601/) (2025)
- [FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model](https://mlsanity.com/paper/2510.10921/) (2025)
- [Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting](https://mlsanity.com/paper/2503.05949/) (2025)
- [Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval](https://mlsanity.com/paper/2505.15877/) (2025)
- [PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning](https://mlsanity.com/paper/2511.04601/) (2025)
- [MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting](https://mlsanity.com/paper/2509.03800/) (2025)
- [Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment](https://mlsanity.com/paper/2603.17655/) (2026)
- [VIRTUE: Visual-Interactive Text-Image Universal Embedder](https://mlsanity.com/paper/2510.00523/) (2025)
- [Identity Card Presentation Attack Detection: A Systematic Review](https://mlsanity.com/paper/2511.06056/) (2025)
- [Revisiting Multi-Task Visual Representation Learning](https://mlsanity.com/paper/2601.13886/) (2026)
- [Text-Conditional JEPA for Learning Semantically Rich Visual Representations](https://mlsanity.com/paper/2605.03245/) (2026)
- [MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook](https://mlsanity.com/paper/2509.14142/) (2025)
- [FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understanding](https://mlsanity.com/paper/2511.14901/) (2025)
- [Peek-a-Boo Reasoning: Contrastive Region Masking in MLLMs](https://mlsanity.com/paper/2512.08976/) (2025)
- [Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families](https://mlsanity.com/paper/2602.15950/) (2026)
- [TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models](https://mlsanity.com/paper/2603.06999/) (2026)
- [Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2605.29776/) (2026)
- [PIG-Nav: Key Insights for Pretrained Image Goal Navigation Models](https://mlsanity.com/paper/2507.17220/) (2025)
- [Token Entropy Regularization for Multi-modal Antenna Affiliation Identification](https://mlsanity.com/paper/2601.21280/) (2026)
- [Xray-Visual Models: Scaling Vision models on Industry Scale Data](https://mlsanity.com/paper/2602.16918/) (2026)
- [Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades](https://mlsanity.com/paper/2603.08028/) (2026)
- [LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment](https://mlsanity.com/paper/2605.08156/) (2026)
- [FAST-GOAL: Fast and Efficient Global-local Object Alignment Learning](https://mlsanity.com/paper/2605.26615/) (2026)
- [From Vision to Text: A Compact Multimodal Approach for Robust, Cross-Domain Presentation Attack Detection on ID Cards](https://mlsanity.com/paper/2606.06966/) (2026)
- [Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks](https://mlsanity.com/paper/2606.19489/) (2026)

## Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels. Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.