mlsanityData to 14 Oct 2026

ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension

arXiv 2204.05991 ↗First posted 12 Apr 2022cs.CV cs.CLcited by 120

Not on the map. Why?

Cited by 37 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain. While large-scale pre-trained models are useful for image classification across domains, it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC. Motivated by the close connection between ReC and CLIP's contrastive pre-training objective, the first component of ReCLIP is a region-scoring method that isolates object proposals via cropping and blurring, and passes them to CLIP. However, through controlled experiments on a synthetic dataset, we find that CLIP is largely incapable of performing spatial reasoning off-the-shelf. Thus, the second component of ReCLIP is a spatial relation resolver that handles several types of spatial relations. We reduce the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game imagery), ReCLIP's relative improvement over supervised ReC models trained on real images is 8%.

Key facts. Cited by 120 papers in the map (data through 14 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

36 of 57 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
24 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  3. 2016Modeling Context in Referring Expressionscited by 1,063
  4. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  6. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  7. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  8. 2021True Few-Shot Learning with Language Modelscited by 344
  9. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  10. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  11. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  12. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  13. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  14. 2021Simple multi-dataset detectioncited by 127
  15. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  16. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  17. 2017Grounding Referring Expressions in Images by Variational Contextcited by 82
  18. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79
  19. 2021Learning to Compose Visual Relationscited by 72
  20. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  21. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  22. 2021Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Groundingcited by 21
  23. 2021OCID-Ref: A 3D Robotic Dataset with Embodied Language for Clutter Scene Groundingcited by 16
  24. 2018Resolving Referring Expressions in Images With Labeled Elementscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 37; 2024: 33; 2025: 20; 2026: 20.

120 papers.

Sort:

By area:

  1. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  2. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  3. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  4. 2022Visual Spatial Reasoningcited by 293
  5. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  6. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  7. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  8. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 151
  9. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  10. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  11. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  12. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
38 more
  1. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  2. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
  3. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  4. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  5. 2023Fine-Grained Visual Promptingcited by 82
  6. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 73
  7. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  8. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  9. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  10. 2024Toward Visual Grounding: A Surveycited by 52
  11. 2023Modular Visual Question Answering via Code Generationcited by 48
  12. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 46
  13. 2023Described Object Detection: Liberating Object Detection with Flexible Expressionscited by 40
  14. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  15. 2024Diffusion Feedback Helps CLIP See Bettercited by 38
  16. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  17. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  18. 2023CLIP-VG: Self-paced Curriculum Adapting of CLIP for Visual Groundingcited by 35
  19. 20254D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Modelscited by 35
  20. 2023Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Trainingcited by 32
  21. 2024HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Groundingcited by 31
  22. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  23. 2024Learning Visual Grounding from Generative Vision and Language Modelcited by 27
  24. 2024Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentationcited by 25
  25. 2024RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognitioncited by 24
  26. 2024Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentationcited by 23
  27. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  28. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  29. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  30. 2023GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detectioncited by 20
  31. 2023Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generationcited by 19
  32. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 19
  33. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Modelscited by 18
  34. 2024HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoningcited by 18
  35. 2024Towards Open-World Grasping with Large Vision-Language Modelscited by 15
  36. 2024Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Modelscited by 15
  37. 2025Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentationcited by 14
  38. 2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 13

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker
  2. 2025RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension
  3. 2026Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks
  4. 2024The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge
  5. 2025KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge
  6. 2025FLORA: Formal Language Model Enables Robust Training-free Zero-shot Object Referring Analysis
  7. 2025Referring Expressions as a Lens into Spatial Language Grounding in Vision-Language Models
  8. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaboration