mlsanityData to 14 Oct 2026

How Far Are We from Generating Missing Modalities with Foundation Models?

arXiv 2506.03530 ↗First posted 4 Jun 2025cs.MM cs.CL cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Multimodal foundation models have demonstrated impressive capabilities across diverse tasks. However, their potential as plug-and-play solutions for missing modality reconstruction remains underexplored. To bridge this gap, we identify and formalize three potential paradigms for missing modality reconstruction, and perform a comprehensive evaluation across these paradigms, covering 42 model variants in terms of reconstruction accuracy and adaptability to downstream tasks. Our analysis reveals that current foundation models often fall short in two critical aspects: (i) fine-grained semantic extraction from the available modalities, and (ii) robust validation of generated modalities. These limitations lead to suboptimal and, at times, misaligned generations. To address these challenges, we propose an agentic framework tailored for missing modality reconstruction. This framework dynamically formulates modality-aware mining strategies based on the input context, facilitating the extraction of richer and more discriminative semantic features. In addition, we introduce a self-refinement mechanism, which iteratively verifies and enhances the quality of generated modalities through internal feedback. Experimental results show that our method reduces FID for missing image reconstruction by at least 14% and MER for missing text reconstruction by at least 10% compared to baselines. Code are released at: https://github.com/Guanzhou-Ke/AFM2.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 58 of its 75 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

58 of 75 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2024GPT-4o System Cardcited by 4,588
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  3. 2014Conditional Generative Adversarial Netscited by 2,162
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  8. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  11. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  12. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  13. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  14. 2025Qwen2.5-Omni Technical Reportcited by 736
  15. 2016Generative Adversarial Text to Image Synthesiscited by 735
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  17. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  18. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  19. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  20. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  21. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  22. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  23. 2024Stable Audio Opencited by 197
  24. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 187
  25. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  26. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  27. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
  28. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  29. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 131
  30. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  31. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
  32. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 87
  33. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 77
  34. 2022M3Care: Learning with Missing Modalities in Multimodal Healthcare Datacited by 72
  35. 2020Deep Partial Multi-View Learningcited by 71
  36. 2023MissDiff: Training Diffusion Models on Tabular Data with Missing Valuescited by 39
  37. 2020Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognitioncited by 36
  38. 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness
  2. 2025Knowledge Bridger: Towards Training-free Missing Modality Completion
  3. 2026Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration
  4. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts
  5. 2023Multimodal Guidance Network for Missing-Modality Inference in Content Moderation
  6. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models
  7. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcity
  8. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models