mlsanityData to 7 Oct 2026

HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding

arXiv 2311.14064 ↗First posted 23 Nov 2023cs.CVcited by 21

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Object categories are typically organized into a multi-granularity taxonomic hierarchy. When classifying categories at different hierarchy levels, traditional uni-modal approaches focus primarily on image features, revealing limitations in complex scenarios. Recent studies integrating Vision-Language Models (VLMs) with class hierarchies have shown promise, yet they fall short of fully exploiting the hierarchical relationships. These efforts are constrained by their inability to perform effectively across varied granularity of categories. To tackle this issue, we propose a novel framework (HGCLIP) that effectively combines CLIP with a deeper exploitation of the Hierarchical class structure via Graph representation learning. We explore constructing the class hierarchy into a graph, with its nodes representing the textual or image features of each category. After passing through a graph encoder, the textual features incorporate hierarchical structure information, while the image features emphasize class-aware features derived from prototypes through the attention mechanism. Our approach demonstrates significant improvements on 11 diverse visual recognition benchmarks. Our codes are fully available at https://github.com/richard-peng-xia/HGCLIP.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 79 references that are in the map. First posted on arXiv 23 Nov 2023.

Builds on

49 of 79 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  2. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Decoupling Zero-Shot Semantic Segmentation15 Dec
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  9. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  10. 2013Describing Textures in the Wildcited by 2,110
  11. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
29 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2022Visual Prompt Tuningcited by 1,674
  3. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  5. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  6. 2019Natural Adversarial Examplescited by 1,401
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  8. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  9. 2015LINE: Large-scale Information Network Embeddingcited by 582
  10. 2020Heterogeneous Graph Transformercited by 581
  11. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  12. 2019Graph Transformer Networkscited by 520
  13. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  14. 2017A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applicationscited by 260
  15. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  16. 2023GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networkscited by 166
  17. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  18. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  19. 2020Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patchescited by 92
  20. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  21. 2020Hierarchical Image Classification using Entailment Cone Embeddingscited by 46
  22. 2022Improving Zero-shot Generalization and Robustness of Multi-modal Modelscited by 44
  23. 2017Fruit recognition from images using deep learningcited by 35
  24. 2018Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embeddingcited by 28
  25. 2022Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classificationcited by 21
  26. 2023LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognitioncited by 15
  27. 2023Learning to Count Isomorphisms with Graph Neural Networkscited by 13
  28. 2021Hierarchical Knowledge Guided Learning for Real-world Retinal Diseases Recognitioncited by 10
  29. 2023Pixel Adapter: A Graph-Based Post-Processing Approach for Scene Text Image Super-Resolutioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 9.

21 papers.

Sort:

By area:

  1. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  2. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  3. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  4. 2024MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimizationcited by 21
  5. 2024TP-DRSeg: Improving Diabetic Retinopathy Lesion Segmentation with Explicit Text-Prompts Assisted SAMcited by 13
  6. 2024Generalizing to Unseen Domains in Diabetic Retinopathy with Disentangled Representationscited by 11
  7. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  8. 2025Delving into Out-of-Distribution Detection with Medical Vision-Language Modelscited by 5
  9. 2025Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoderscited by 5
  10. 2025Generalizable Prompt Learning of CLIP: A Brief Overviewcited by 3
  11. 2025Hierarchical Material Recognition from Local Appearancecited by 3
  12. 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Modelscited by 3
9 more
  1. 2024Diffusion Model Driven Test-Time Image Adaptation for Robust Skin Lesion Classificationcited by 2
  2. 2026CLEAR-HPV: Interpretable concept discovery for human-papillomavirus-associated morphology in whole-slide histologycited by 0
  3. 2026From Semantics to Pixels: Coarse-to-Fine Masked Autoencoders for Hierarchical Visual Understandingcited by 0
  4. 2026HELM: Hierarchical and Explicit Label Modeling with Graph Learning for Multi-Label Image Classificationcited by 0
  5. 2026RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMscited by 0
  6. 2026MAPLE: Multi-Path Adaptive Propagation with Level-Aware Embeddings for Hierarchical Multi-Label Image Classificationcited by 0
  7. 2026Beyond Flat Labels: Level-Restricted Contrastive Learning for Hierarchical Fine-Grained Vision Classificationcited by 0
  8. 2026Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Modelscited by 0
  9. 2026Hierarchical Prompt Learning for Hyperbolic Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
  2. 2023DesCo: Learning Object Recognition with Rich Language Descriptions
  3. 2024GraphVL: Graph-Enhanced Semantic Modeling via Vision-Language Models for Generalized Class Discovery
  4. 2022Visual Classification via Description from Large Language Models
  5. 2023ChatGPT-Powered Hierarchical Comparisons for Image Classification
  6. 2024Enhancing Visual Classification using Comparative Descriptors
  7. 2026Explaining, Verifying, and Aligning Semantic Hierarchies in Vision-Language Model Embeddings
  8. 2025Hierarchy-Aware Fine-Tuning of Vision-Language Models