mlsanityData to 7 Oct 2026

RegionCLIP: Region-based Language-Image Pretraining

arXiv 2112.09106 ↗First posted 16 Dec 2021cs.CV cs.AI cs.LGcited by 612

Not on the map. Why?

Cited by 157 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive language-image pretraining (CLIP) using image-text pairs has achieved impressive results on image classification in both zero-shot and transfer learning settings. However, we show that directly applying such models to recognize image regions for object detection leads to poor performance due to a domain shift: CLIP was trained to match an image as a whole to a text description, without capturing the fine-grained alignment between image regions and text spans. To mitigate this issue, we propose a new method called RegionCLIP that significantly extends CLIP to learn region-level visual representations, thus enabling fine-grained alignment between image regions and textual concepts. Our method leverages a CLIP model to match image regions with template captions and then pretrains our model to align these region-text pairs in the feature space. When transferring our pretrained model to the open-vocabulary object detection tasks, our method significantly outperforms the state of the art by 3.8 AP50 and 2.2 AP for novel categories on COCO and LVIS datasets, respectively. Moreoever, the learned region representations support zero-shot inference for object detection, showing promising results on both COCO and LVIS datasets. Our code is available at https://github.com/microsoft/RegionCLIP.

Key facts. Cited by 612 papers in the map (data through 7 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

49 of 71 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
37 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,573
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  13. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  16. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  17. 2017Focal Loss for Dense Object Detectioncited by 749
  18. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  19. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  20. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  21. 2020Meta Pseudo Labelscited by 372
  22. 2020Rethinking Pre-training and Self-trainingcited by 352
  23. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  24. 2021End-to-End Semi-Supervised Object Detection with Soft Teachercited by 315
  25. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  26. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  27. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  28. 2018Zero-Shot Object Detectioncited by 189
  29. 2015Learning Visual Features from Large Weakly Supervised Datacited by 165
  30. 2021Efficient Visual Pretraining with Contrastive Detectioncited by 150
  31. 2021Probabilistic two-stage detectioncited by 130
  32. 2020Learning Visual Representations with Caption Annotationscited by 123
  33. 2015Webly Supervised Learning of Convolutional Networkscited by 95
  34. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  35. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  36. 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
  37. 2017R-FCN-3000 at 30fps: Decoupling Detection and Classificationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 39; 2023: 177; 2024: 153; 2025: 140; 2026: 103.

612 papers.

Sort:

By area:

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 974
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  6. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  7. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  8. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  9. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  10. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  11. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  12. 2023Scaling Open-Vocabulary Object Detectioncited by 335
38 more
  1. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
  2. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  4. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  5. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  6. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  7. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  8. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 270
  9. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  10. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  11. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  12. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  13. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  14. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  15. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  16. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  17. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  18. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  19. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  20. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  21. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  22. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  23. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  24. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  25. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130
  26. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  27. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  28. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  29. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  30. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  31. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  32. 2023Contextual Object Detection with Multimodal Large Language Modelscited by 100
  33. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  34. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  35. 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 95
  36. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  37. 2024T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergycited by 94
  38. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93

50 most cited of 612 All 612 (JSON)