mlsanityData to 14 Oct 2026

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

arXiv 2112.01518 ↗First posted 2 Dec 2021cs.CV cs.AI cs.LGcited by 532

Not on the map. Why?

Cited by 165 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. By further using the contextual information from the image to prompt the language model, we are able to facilitate our model to better exploit the pre-trained knowledge. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones including both CLIP models and ImageNet pre-trained models. Extensive experiments demonstrate the superior performance of our methods on semantic segmentation, object detection, and instance segmentation tasks. Code is available at https://github.com/raoyongming/DenseCLIP

Key facts. Cited by 532 papers in the map (data through 14 Oct 2026). Builds on 53 of its 60 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

53 of 60 references.

References · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  12. 2017Mask R-CNNcited by 7,277
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2016Pyramid Scene Parsing Networkcited by 2,479
  10. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  13. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  14. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  15. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  16. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  17. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  18. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  19. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  20. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  21. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  22. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  23. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  24. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  25. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  26. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
  27. 2019Panoptic Feature Pyramid Networkscited by 645
  28. 2021ImageNet-21K Pretraining for the Massescited by 613
  29. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
  30. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  31. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  32. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  33. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  34. 2020Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learningcited by 297
  35. 2018Context Encoding for Semantic Segmentationcited by 290
  36. 2021Global Filter Networks for Image Classificationcited by 260
  37. 2016Segmentation from Natural Language Expressionscited by 257
  38. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 58; 2023: 176; 2024: 146; 2025: 98; 2026: 53.

532 papers.

Sort:

By area:

  1. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  2. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
  3. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  4. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  5. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  6. 2023Segment Everything Everywhere All at Oncecited by 548
  7. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  8. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  9. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  10. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  11. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  12. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
38 more
  1. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  2. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  3. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  4. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  5. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  6. 2023AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detectioncited by 271
  7. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  8. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  9. 2023A Cookbook of Self-Supervised Learningcited by 250
  10. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  11. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  12. 2023CLIP-Driven Universal Model for Organ Segmentation and Tumor Detectioncited by 224
  13. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  14. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  15. 2022CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labelscited by 181
  16. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  17. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 181
  18. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 179
  19. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  20. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  21. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  22. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  23. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  24. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  25. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 123
  26. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  27. 2022PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Modelscited by 118
  28. 2022ReCo: Retrieve and Co-segment for Zero-shot Transfercited by 117
  29. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  30. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  31. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  32. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  33. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 96
  34. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 96
  35. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  36. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  37. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  38. 2023Vision-Language Models in Remote Sensing: Current Progress and Future Trendscited by 93

50 most cited of 532 All 532 (JSON)