mlsanityData to 7 Oct 2026

AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning

arXiv 2605.24816 ↗First posted 24 May 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Deploying multimodal systems in real-world environments often entails handling modality-missing scenarios, where one or more modalities are unavailable. While recent studies address this challenge for the general Multimodal Transformer (MT) architecture via prompt tuning, we identify a fundamental limitation in these methods: the Implicit Modality-Reduction bottleneck. By conditioning prompts solely on the observed modalities, they inadvertently restrict the reasoning scope of MTs to the modality-reduced subspace, cutting off access to the latent information sources of the missing modalities. To overcome this limitation, we propose AOEPT, which pioneers a novel modal-contextualized prompting fashion. Specifically, we introduce lightweight Modal-Contextualized Prompts (MCPs) that distill global modality-wise priors from training data, serving as latent repositories of the information sources for missing modalities. Conditioned on the remaining modalities, these MCPs are instantiated into instance-aware prompts that selectively augment missing-modality information for each sample, thereby restoring the reasoning scope of MTs beyond the observed-modality-only subspace. Experiments across various multimodal benchmarks and backbones confirm the strong performance of AOEPT, with minimal computational overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

29 of 46 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  7. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
  10. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
  11. 2017Gated Multimodal Units for Information Fusioncited by 244
  12. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
17 more
  1. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  2. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 159
  3. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  4. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  5. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  6. 2024Revisiting the Power of Prompt for Visual Tuningcited by 32
  7. 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 22
  8. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  9. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17
  10. 2023Towards Robust Multimodal Prompting With Missing Modalitiescited by 16
  11. 2025Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learningcited by 15
  12. 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Modelscited by 12
  13. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Modelscited by 9
  14. 2025Synergistic Prompting for Robust Visual Recognition with Missing Modalitiescited by 8
  15. 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
  16. 2026Nip Rumors in the Bud: Retrieval-Guided Topic-Level Adaptation for Test-Time Fake News Video Detectioncited by 1
  17. 2026Shedding the Facades, Connecting the Domains: Detecting Shifting Multimodal Hate Video with Test-Time Adaptationcited by 1

Built on by

Not cited yet.