mlsanityData to 7 Oct 2026

GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph

arXiv 2309.13625 ↗First posted 24 Sep 2023cs.CV cs.AIcited by 69

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Adapter-style efficient transfer learning (ETL) has shown excellent performance in the tuning of vision-language models (VLMs) under the low-data regime, where only a few additional parameters are introduced to excavate the task-specific knowledge based on the general and powerful representation of VLMs. However, most adapter-style works face two limitations: (i) modeling task-specific knowledge with a single modality only; and (ii) overlooking the exploitation of the inter-class relationships in downstream tasks, thereby leading to sub-optimal solutions. To mitigate that, we propose an effective adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual adapter by explicitly modeling the dual-modality structure knowledge (i.e., the correlation of different semantics/classes in textual and visual modalities) with a dual knowledge graph. In particular, the dual knowledge graph is established with two sub-graphs, i.e., a textual knowledge sub-graph, and a visual knowledge sub-graph, where the nodes and edges represent the semantics/classes and their correlations in two modalities, respectively. This enables the textual feature of each prompt to leverage the task-specific structure knowledge from both textual and visual modalities, yielding a more effective classifier for downstream tasks. Extensive experimental results on 11 benchmark datasets reveal that our GraphAdapter significantly outperforms previous adapter-based methods. The code will be released at https://github.com/lixinustc/GraphAdapter

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 92 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

69 of 92 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr
  2. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 66

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
38 more
  1. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2013Describing Textures in the Wildcited by 2,110
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  11. 2019Natural Adversarial Examplescited by 1,401
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  13. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  14. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  15. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  16. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  17. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  18. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  19. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  20. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 530
  21. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  22. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  23. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  24. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  25. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  26. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  27. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  28. 2022Prompt Distribution Learningcited by 235
  29. 2022Knowledge Distillation from A Stronger Teachercited by 204
  30. 2023DepGraph: Towards Any Structural Pruningcited by 202
  31. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  32. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  33. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  34. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  35. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  36. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  37. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  38. 2022Task Residual for Tuning Vision-Language Modelscited by 122

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 29; 2025: 28; 2026: 8.

69 papers.

Sort:

By area:

  1. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answeringcited by 217
  2. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  3. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  4. 2024Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Imagescited by 69
  5. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  6. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
  7. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  8. 2024KVQ: Kwai Video Quality Assessment for Short-form Videoscited by 38
  9. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  10. 2024Robust Calibration of Large Vision-Language Adapterscited by 28
  11. 2024SeD: Semantic-Aware Discriminator for Image Super-Resolutioncited by 26
  12. 2024DOTA: Distributional Test-Time Adaptation of Vision-Language Modelscited by 22
38 more
  1. 2025Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoningcited by 19
  2. 2024Selective Vision-Language Subspace Projection for Few-shot CLIPcited by 18
  3. 2024BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrappingcited by 17
  4. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  5. 2023Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Modelscited by 13
  6. 2025COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptationcited by 13
  7. 2025SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Promptingcited by 11
  8. 2025Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapterscited by 9
  9. 2024Enhancing Vision-Language Few-Shot Adaptation with Negative Learningcited by 8
  10. 2024Noise-powered Multi-modal Knowledge Graph Representation Frameworkcited by 7
  11. 2025Unifying Text Semantics and Graph Structures for Temporal Text-attributed Graphs with Large Language Modelscited by 6
  12. 2025CATCH: A Modular Cross-domain Adaptive Template with Hookcited by 6
  13. 2024A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learningcited by 5
  14. 2024Not All Prompts Are Secure: A Switchable Backdoor Attack Against Pre-trained Vision Transformerscited by 5
  15. 2024TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaborationcited by 5
  16. 2024Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Modelscited by 4
  17. 2024Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Modelcited by 4
  18. 2024Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approachcited by 4
  19. 2024Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptationcited by 4
  20. 2024Pin-Tuning: Parameter-Efficient In-Context Tuning for Few-Shot Molecular Property Predictioncited by 4
  21. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  22. 2025Self-evolved Imitation Learning in Simulated Worldcited by 4
  23. 2024Conditional Prototype Rectification Prompt Learningcited by 3
  24. 2024HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adaptercited by 3
  25. 2025Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervisioncited by 3
  26. 2025Schema Generation for Large Knowledge Graphs Using Large Language Modelscited by 3
  27. 2025BMIP: Bi-directional Modality Interaction Prompt Learning for VLMcited by 2
  28. 2025DesCLIP: Robust Continual Learning via General Attribute Descriptions for VLM-Based Visual Recognitioncited by 2
  29. 2025Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Modelscited by 2
  30. 2026Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approachcited by 2
  31. 2024Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision Transformers for High-Level Image Classificationcited by 1
  32. 2024wav2graph: A Framework for Supervised Learning Knowledge Graph from Speechcited by 1
  33. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  34. 2024Cross-Modal Few-Shot Learning with Second-Order Neural Ordinary Differential Equationscited by 1
  35. 2025CLIP-DQA: Blindly Evaluating Dehazed Images from Global and Local Perspectives Using CLIPcited by 1
  36. 2025Generalizable Vision-Language Few-Shot Adaptation with Predictive Prompts and Negative Learningcited by 1
  37. 2025Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adaptercited by 1
  38. 2025COLA: Context-aware Language-driven Test-time Adaptationcited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models
  2. 2025Transferable Model-agnostic Vision-Language Model Adaptation for Efficient Weak-to-Strong Generalization
  3. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models
  4. 2025RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models
  5. 2023Towards Versatile and Efficient Visual Knowledge Integration into Pre-trained Language Models with Cross-Modal Adapters
  6. 2026SmoGVLM: A Small, Graph-enhanced Vision-Language Model
  7. 2025Dynamic Knowledge Integration for Enhanced Vision-Language Reasoning
  8. 2026GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning