mlsanityData to 7 Oct 2026

UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding

arXiv 2401.06397 ↗First posted 12 Jan 2024cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level as well as pixel-level captions and tags. Accordingly, we develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities across different levels of detail. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks. We believe that UMG-CLIP represents a valuable advancement in vision-language foundation models. The code is available at https://github.com/lygsbw/UMG-CLIP.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 78 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

59 of 78 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentation23 Feb
  2. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model29 Dec
  3. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2023Segment Anythingcited by 8,158
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
38 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  4. 2019Natural Adversarial Examplescited by 1,401
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2021Grounded Language-Image Pre-trainingcited by 1,147
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  11. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  12. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  13. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  14. 2022Language-driven Semantic Segmentationcited by 711
  15. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  16. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  17. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  18. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  19. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  20. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  21. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  22. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 371
  23. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  24. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  25. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  26. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  27. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  28. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  29. 2019Zero-Shot Semantic Segmentationcited by 245
  30. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  31. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 199
  32. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  33. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  34. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  35. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  36. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  37. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  38. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  2. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  3. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  4. 2024FALIP: Visual Prompt as Foveal Attention Boosts CLIP Zero-Shot Performancecited by 7
  5. 2025FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Textcited by 7
  6. 2024Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Modelscited by 3
  7. 2025SeLIP: Similarity Enhanced Contrastive Language Image Pretraining for Multi-modal Head MRIcited by 2
  8. 2025un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIPcited by 2
  9. 2025MindVL: Towards Efficient and Effective Training of Multimodal Large Language Models on Ascend NPUscited by 2
  10. 2026Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentationcited by 2
  11. 2025GranViT: A Fine-Grained Vision Model With Autoregressive Perception For MLLMscited by 1
  12. 2026Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomographycited by 1
1 more
  1. 2026Hierarchical Pre-Training of Vision Encoders with Large Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TULIP: Towards Unified Language-Image Pretraining
  2. 2025Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
  3. 2025The Finer the Better: Towards Granular-aware Open-set Domain Generalization
  4. 2025From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection
  5. 2025PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning
  6. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  7. 2025HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Models
  8. 2025FG-CLIP: Fine-Grained Visual and Textual Alignment