# Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities

arXiv: https://arxiv.org/abs/2606.30597 · HTML: https://mlsanity.com/paper/2606.30597/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2303.12501/) (2023)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Multi-modal Dense Video Captioning](https://mlsanity.com/paper/2003.07758/) (2020)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluation](https://mlsanity.com/paper/2303.12112/) (2023)
- [Single-Model and Any-Modality for Video Object Tracking](https://mlsanity.com/paper/2311.15851/) (2023)
- [DePT: Decoupled Prompt Tuning](https://mlsanity.com/paper/2309.07439/) (2023)
- [MMRL: Multi-Modal Representation Learning for Vision-Language Models](https://mlsanity.com/paper/2503.08497/) (2025)
- [Continual Multimodal Contrastive Learning](https://mlsanity.com/paper/2503.14963/) (2025)

## Abstract (arXiv)

Large-scale multimodal models (LMMs) have achieved superior performance in visual recognition by synergizing information across diverse, massive-scale paired modalities. In real-world scenarios, however, missing-modality inputs are ubiquitous, causing models optimized for modality-complete data to exhibit precipitous performance degradation. Existing research has introduced prompt learning to mitigate this issue, typically by generating dynamic prompts from instance-level features, regardless of whether the input modalities are complete or partially absent. However, such input-conditioned strategies are hindered by the escalating unreliability of instance-level features; as higher missing rates increase the proportion of incomplete modalities, the resulting instability in prompt learning limits the model's performance. To address this limitation, we hypothesize that learnable latent prompts themselves encapsulate stable, modality-intrinsic priors that are decoupled from corrupted inputs. Consequently, we propose a novel paradigm: Learning from Reliable Latent Prompts. Unlike prior methods, we model input-agnostic learnable prompts as stable latent anchors that enable robust guidance and effective cross-modal knowledge compensation, even under extreme missing rates (e.g., 90%). Empirical results across three benchmark datasets demonstrate that our "learn-from-latent-prompts" approach achieves state-of-the-art performance across a wide range of missing-modality scenarios. Extensive experiments further confirm the effectiveness of this paradigm in providing a robust solution to the missing-modality problem.