mlsanityData to 14 Oct 2026

HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter

arXiv 2410.07854 ↗First posted 10 Oct 2024cs.CV cs.MMcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Adapter-based tuning methods have shown significant potential in transferring knowledge from pre-trained Vision-Language Models to the downstream tasks. However, after reviewing existing adapters, we find they generally fail to fully explore the interactions between different modalities in constructing task-specific knowledge. Also, existing works usually only focus on similarity matching between positive text prompts, making it challenging to distinguish the classes with high similar visual contents. To address these issues, in this paper, we propose a novel Heterogeneous Graph Adapter to achieve tuning VLMs for the downstream tasks. To be specific, we first construct a unified heterogeneous graph mode, which contains i) visual nodes, positive text nodes and negative text nodes, and ii) several types of edge connections to comprehensively model the intra-modality, inter-modality and inter-class structure knowledge together. Next, we employ a specific Heterogeneous Graph Neural Network to excavate multi-modality structure knowledge for adapting both visual and textual features for the downstream tasks. Finally, after HeGraphAdapter, we construct both text-based and visual-based classifiers simultaneously to comprehensively enhance the performance of the CLIP model. Experimental results on 11 benchmark datasets demonstrate the effectiveness and benefits of the proposed HeGraphAdapter.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 57 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

41 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph24 Sep

Other references · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
28 more
  1. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  4. 2019Natural Adversarial Examplescited by 1,408
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  8. 2019Heterogeneous Graph Attention Networkcited by 734
  9. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  10. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  11. 2020MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embeddingcited by 304
  12. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  13. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 141
  14. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  15. 2017Heterogeneous Information Network Embedding for Recommendationcited by 133
  16. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  17. 2020A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sourcescited by 126
  18. 2022Task Residual for Tuning Vision-Language Modelscited by 123
  19. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 119
  20. 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
  21. 2019An Attention-based Graph Neural Network for Heterogeneous Structural Learningcited by 71
  22. 2022Simple and Efficient Heterogeneous Graph Neural Networkcited by 71
  23. 2022Zero-Shot Temporal Action Detection via Vision-Language Promptingcited by 65
  24. 2022Dual Modality Prompt Tuning for Vision-Language Pre-Trained Modelcited by 52
  25. 2024AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learningcited by 22
  26. 2020Task-Adaptive Negative Envision for Few-Shot Open-Set Recognitioncited by 14
  27. 2023YUAN 2.0: A Large Language Model with Localized Filtering-based Attentioncited by 9
  28. 2024GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modelingcited by 3
  2. 2025Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adaptercited by 1
  3. 2026MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SmoGVLM: A Small, Graph-enhanced Vision-Language Model
  2. 2023p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models
  3. 2026One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models
  4. 2023Towards Versatile and Efficient Visual Knowledge Integration into Pre-trained Language Models with Cross-Modal Adapters
  5. 2026VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings
  6. 2026HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models
  7. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning
  8. 2026GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning