mlsanityData to 7 Oct 2026

Modular and Parameter-Efficient Multimodal Fusion with Prompting

arXiv 2203.08055 ↗First posted 15 Mar 2022cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Recent research has made impressive progress in large-scale multimodal pre-training. In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than finetuning. In this paper, we propose to use prompt vectors to align the modalities. Our method achieves comparable performance to several other multimodal fusion methods in low-resource settings. We further show that our method is modular and parameter-efficient for processing tasks involving two or more data modalities.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

42 of 50 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
30 more
  1. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  13. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  14. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  16. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  17. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  18. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  19. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  20. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  21. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  22. 2021How Many Data Points is a Prompt Worth?cited by 197
  23. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  24. 2019Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)cited by 173
  25. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  26. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  27. 2021M6: A Chinese Multimodal Pretrainercited by 91
  28. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  29. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  30. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 13; 2024: 11; 2025: 5; 2026: 4.

35 papers.

Sort:

By area:

  1. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  2. 2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
  3. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  4. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  5. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  6. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  7. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Promptscited by 19
  8. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  9. 2024M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learningcited by 16
  10. 2023Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognitioncited by 14
  11. 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Modelscited by 12
  12. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
23 more
  1. 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
  2. 2024OpenMU: Your Swiss Army Knife for Music Understandingcited by 9
  3. 2024Skipping Computations in Multimodal LLMscited by 7
  4. 2024MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusioncited by 5
  5. 2022Learning to translate by learning to communicatecited by 4
  6. 2023BLIAM: Literature-based Data Synthesis for Synergistic Drug Combination Predictioncited by 3
  7. 2024Multi-rater Prompting for Ambiguous Medical Image Segmentationcited by 3
  8. 2024Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQAcited by 3
  9. 2025ConceptFormer: Towards Efficient Use of Knowledge-Graph Embeddings in Large Language Modelscited by 3
  10. 2025Multimodal Representation Learning and Fusioncited by 3
  11. 2023Modality Plug-and-Play: Elastic Modality Adaptation in Multimodal LLMs for Embodied AIcited by 2
  12. 2024Memory-Inspired Temporal Prompt Interaction for Text-Image Classificationcited by 1
  13. 2024Contrast then Memorize: Semantic Neighbor Retrieval-Enhanced Inductive Multimodal Knowledge Graph Completioncited by 1
  14. 2026AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMscited by 1
  15. 2023Conditional Prompt Tuning for Multimodal Fusioncited by 0
  16. 2024FSMR: A Feature Swapping Multi-modal Reasoning Approach with Joint Textual and Visual Cluescited by 0
  17. 2024Robust Latent Representation Tuning for Image-text Classificationcited by 0
  18. 2025EPIC: Efficient Prompt Interaction for Text-Image Classificationcited by 0
  19. 2025FusionAdapter for Few-Shot Relation Learning in Multimodal Knowledge Graphscited by 0
  20. 2025Teaching Sarcasm: Few-Shot Multimodal Sarcasm Detection via Distillation to a Parameter-Efficient Studentcited by 0
  21. 2026E2PL: Effective and Efficient Prompt Learning for Incomplete Multi-view Multi-Label Class Incremental Learningcited by 0
  22. 2026Contrastive Mixed Prompt Learning for Incomplete Multimodal Sentiment Analysis with Unseen Modality Combinationcited by 0
  23. 2026Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysiscited by 0