mlsanityData to 7 Oct 2026

Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning

arXiv 2405.05615 ↗First posted 9 May 2024cs.CV cs.CL cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Current solutions for efficiently constructing large vision-language (VL) models follow a two-step paradigm: projecting the output of pre-trained vision encoders to the input space of pre-trained language models as visual prompts; and then transferring the models to downstream VL tasks via end-to-end parameter-efficient fine-tuning (PEFT). However, this paradigm still exhibits inefficiency since it significantly increases the input length of the language models. In this paper, in contrast to integrating visual prompts into inputs, we regard visual prompts as additional knowledge that facilitates language models in addressing tasks associated with visual information. Motivated by the finding that Feed-Forward Network (FFN) of language models acts as "key-value memory", we introduce a novel approach termed memory-space visual prompting (MemVP), wherein visual prompts are concatenated with the weights of FFN for visual knowledge injection. Experimental results across various VL tasks and language models reveal that MemVP significantly reduces the training time and inference latency of the finetuned VL models and surpasses the performance of previous PEFT methods. Code: https://github.com/JieShibo/MemVP

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 47 of its 53 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

47 of 53 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
35 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2021Scaling Vision Transformerscited by 993
  7. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  12. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  15. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  16. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  17. 2017Learning multiple visual domains with residual adapterscited by 553
  18. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  19. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  20. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  21. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  22. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  23. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  24. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  25. 2022Neural Prompt Searchcited by 133
  26. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  27. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  28. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  29. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  30. 2023UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modelingcited by 40
  31. 2023VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Controlcited by 21
  32. 2022Neural Knowledge Bank for Pretrained Transformerscited by 16
  33. 2024PanGu-π Pro:Rethinking Optimization and Architecture for Tiny Language Modelscited by 14
  34. 2022HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Taskscited by 12
  35. 2023MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answeringcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 5.

20 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  3. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  4. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  5. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  6. 2024Visual Perception by Large Language Model's Weightscited by 19
  7. 2024Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMscited by 17
  8. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Modelscited by 13
  9. 2025CyberV: Cybernetics for Test-time Scaling in Video Understandingcited by 8
  10. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  11. 2024Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuningcited by 5
  12. 2026UAOR: Uncertainty-aware Observation Reinjection for Vision-Language-Action Modelscited by 5
8 more
  1. 2025HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Modelscited by 4
  2. 2024ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuningcited by 2
  3. 2026Enhancing Zero-shot Personalized Image Aesthetics Assessment with Profile-aware Multimodal LLMcited by 2
  4. 2026SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Modelscited by 2
  5. 2025Dynamic Embedding of Hierarchical Visual Features for Efficient Vision-Language Fine-Tuningcited by 1
  6. 2025Remodeling Semantic Relationships in Vision-Language Fine-Tuningcited by 0
  7. 2026LoReC: Rethinking Large Language Models for Graph Data Analysiscited by 0
  8. 2026BiomedAP: A Vision-Informed Dual-Anchor Framework with Gated Cross-Modal Fusion for Robust Medical Vision-Language Adaptationcited by 0