mlsanityData to 14 Oct 2026

A Critical Look at the Current Usage of Foundation Model for Dense Recognition Task

arXiv 2307.02862 ↗First posted 6 Jul 2023cs.CVcited by 1

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

In recent years large model trained on huge amount of cross-modality data, which is usually be termed as foundation model, achieves conspicuous accomplishment in many fields, such as image recognition and generation. Though achieving great success in their original application case, it is still unclear whether those foundation models can be applied to other different downstream tasks. In this paper, we conduct a short survey on the current methods for discriminative dense recognition tasks, which are built on the pretrained foundation model. And we also provide some preliminary experimental analysis of an existing open-vocabulary segmentation method based on Stable Diffusion, which indicates the current way of deploying diffusion model for segmentation is not optimal. This aims to provide insights for future research on adopting foundation model for downstream task.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 40 of its 44 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

40 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models8 Mar
  2. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Segment Anythingcited by 8,234
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
26 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2020In Search of Lost Domain Generalizationcited by 1,054
  5. 2020Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptationcited by 848
  6. 2017Maximum Classifier Discrepancy for Unsupervised Domain Adaptationcited by 683
  7. 2021Extract Free Dense Labels from CLIPcited by 583
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  9. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 353
  10. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  11. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 306
  12. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  13. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 269
  14. 2022Generalized Category Discoverycited by 211
  15. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  16. 2021Generalized Source-free Domain Adaptationcited by 175
  17. 2021Open-World Semi-Supervised Learningcited by 165
  18. 2018Classification-Reconstruction Learning for Open-Set Recognitioncited by 146
  19. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 143
  20. 2019C2AE: Class Conditioned Auto-Encoder for Open-set Recognitioncited by 142
  21. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  22. 2022CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentationcited by 118
  23. 2021Model-Based Domain Generalizationcited by 104
  24. 2023FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentationcited by 95
  25. 2021Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Datacited by 55
  26. 2022OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learningcited by 37

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Suppress Content Shift: Better Diffusion Features via Off-the-Shelf Generation Techniquescited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Image Segmentation in Foundation Model Era: A Survey
  2. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
  3. 2026Understanding Why Foundation Models Work for Diffusion-Generated Image Detection
  4. 2023Open-vocabulary Object Segmentation with Diffusion Models
  5. 2024Not All Diffusion Model Activations Have Been Evaluated as Discriminative Features
  6. 2024Diffusion Features to Bridge Domain Gap for Semantic Segmentation
  7. 2024Explore In-Context Segmentation via Latent Diffusion Models
  8. 2026Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe