mlsanityData to 7 Oct 2026

Contrastive Localized Language-Image Pre-Training

arXiv 2410.02746 ↗First posted 3 Oct 2024cs.CV cs.LGcited by 33

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels. Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 45 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

39 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023SILC: Improving Vision Language Pretraining with Self-Distillation20 Oct
  2. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captions11 Oct

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2017Mask R-CNNcited by 7,259
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
25 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2016Modeling Context in Referring Expressionscited by 1,054
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  8. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  10. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  11. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  12. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  13. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  14. 2023Data Filtering Networkscited by 247
  15. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  16. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  17. 2023Improving CLIP Training with Language Rewritescited by 216
  18. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  19. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  20. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  21. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  22. 2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
  23. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  24. 2024Improving fine-grained understanding in image-text pre-trainingcited by 44
  25. 2023MOFI: Learning Image Representations from Noisy Entity Annotated Imagescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 16.

33 papers.

Sort:

By area:

  1. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  2. 2025Scaling Vision Pre-Training to 4K Resolutioncited by 26
  3. 2025Argus: Vision-Centric Reasoning with Grounded Chain-of-Thoughtcited by 26
  4. 2025GOAL: Global-local Object Alignment Learningcited by 17
  5. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
  6. 2025OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learningcited by 15
  7. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  8. 2025Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splattingcited by 5
  9. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  10. 2025MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reportingcited by 4
  11. 2025PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learningcited by 4
  12. 2026Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignmentcited by 4
21 more
  1. 2025VIRTUE: Visual-Interactive Text-Image Universal Embeddercited by 2
  2. 2025Identity Card Presentation Attack Detection: A Systematic Reviewcited by 2
  3. 2026Revisiting Multi-Task Visual Representation Learningcited by 2
  4. 2026Text-Conditional JEPA for Learning Semantically Rich Visual Representationscited by 2
  5. 2025MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlookcited by 1
  6. 2025FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understandingcited by 1
  7. 2025Peek-a-Boo Reasoning: Contrastive Region Masking in MLLMscited by 1
  8. 2026Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Familiescited by 1
  9. 2026TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Modelscited by 1
  10. 2026Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learningcited by 1
  11. 2025PIG-Nav: Key Insights for Pretrained Image Goal Navigation Modelscited by 0
  12. 2026Token Entropy Regularization for Multi-modal Antenna Affiliation Identificationcited by 0
  13. 2026Xray-Visual Models: Scaling Vision models on Industry Scale Datacited by 0
  14. 2026Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascadescited by 0
  15. 2026LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignmentcited by 0
  16. 2026FAST-GOAL: Fast and Efficient Global-local Object Alignment Learningcited by 0
  17. 2026From Vision to Text: A Compact Multimodal Approach for Robust, Cross-Domain Presentation Attack Detection on ID Cardscited by 0
  18. 2026Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottleneckscited by 0
  19. 2026SigLIP-HD by Fine-to-Coarse Supervisioncited by 0
  20. 2026Local Multimodal Music Alignment from Global Supervisioncited by 0
  21. 2026Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  2. 2023Improved baselines for vision-language pre-training
  3. 2025HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Models
  4. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  5. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-training
  6. 2024Diffusion Feedback Helps CLIP See Better
  7. 2026CLIMP: Contrastive Language-Image Mamba Pretraining
  8. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models