mlsanityData to 14 Oct 2026

Buffer replay enhances the robustness of multimodal learning under missing-modality

arXiv 2511.23070 ↗First posted 28 Nov 2025cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Missing modalities consistently lead to significant performance degradation in multimodal models. Existing approaches either synthesize missing modalities at high computational cost or apply prompt-based fine-tuning that relies only on adjacent-layer features and overlooks long-distance contextual information, which may offer additional tolerance to errors when one or more modalities are missing. To address this, we introduce REplay Prompting (REP): (1) construct modality-wise feature buffers via a residual bypass to cache early-layer representations and replay them in deeper layers, mitigating information loss as network depth increases; (2) employ a private-shared feature decoupling strategy, where private buffers preserve modality-specific signals and shared buffers encode cross-modal semantics; and (3) design a task-aware dynamic initialization mechanism to configure these buffers differently, improving stability and generalization under diverse missing-modality conditions. Experiments on vision-language, vision-language-audio, and temporal multimodal benchmarks demonstrate that REP consistently outperforms prior methods under both single- and multi-modality missing scenarios, while introducing only negligible parameter overhead. These results establish REP as a lightweight and effective paradigm for robust multimodal learning in challenging missing-modality environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 28 Nov 2025.

Builds on

30 of 35 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  5. 2022Visual Prompt Tuningcited by 1,681
  6. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  7. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  8. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  9. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  10. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  11. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  12. 2017Gated Multimodal Units for Information Fusioncited by 248
18 more
  1. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  2. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  3. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 194
  4. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  5. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
  6. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  7. 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
  8. 2023Read-only Prompt Optimization for Vision-Language Few-shot Learningcited by 58
  9. 2023DePT: Decoupled Prompt Tuningcited by 58
  10. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  11. 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 33
  12. 2024Towards Robust Multimodal Sentiment Analysis with Incomplete Datacited by 27
  13. 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26
  14. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  15. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 18
  16. 2023Towards Robust Multimodal Prompting With Missing Modalitiescited by 16
  17. 2025I3-MRec: Invariant Learning with Information Bottleneck for Incomplete Modality Recommendationcited by 2
  18. 2025Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalitiescited by 2

Built on by

Not cited yet.