mlsanityData to 7 Oct 2026

Learning Unseen Modality Interaction

arXiv 2306.12795 ↗First posted 22 Jun 2023cs.CV cs.LG cs.MMcited by 12

Not on the map. Why?

Abstract (arXiv)

Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. In this paper, we challenge this modality-complete assumption for multimodal learning and instead strive for generalization to unseen modality combinations during inference. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved. This allows the information to be accumulated with a simple summation operation across available modalities. To reduce overfitting to less discriminative modality combinations during training, we further improve the model learning with pseudo-supervision indicating the reliability of a modality's prediction. We demonstrate that our approach is effective for diverse tasks and modalities by evaluating it for multimodal video classification, robot state regression, and multimedia retrieval. Project website: https://xiaobai1217.github.io/Unseen-Modality-Interaction/.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 31 of its 46 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

31 of 46 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2018SlowFast Networks for Video Recognitioncited by 1,862
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2021Video Swin Transformercited by 1,090
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  11. 2021AST: Audio Spectrogram Transformercited by 634
  12. 2020Rescaling Egocentric Visioncited by 529
19 more
  1. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  2. 2020Multi-modal Transformer for Video Retrievalcited by 395
  3. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  4. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  5. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  6. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  7. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  8. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  9. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  10. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  11. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  12. 2019Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQAcited by 107
  13. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  14. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 80
  15. 2021Robust Audio-Visual Instance Discriminationcited by 80
  16. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  17. 2022Audio-Adaptive Activity Recognition Across Video Domainscited by 35
  18. 2022Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalitiescited by 28
  19. 2023Zorro: the masked multimodal transformercited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 2.

12 papers.

  1. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  2. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 29
  3. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  4. 2025Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalitiescited by 10
  5. 2025ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Modelcited by 6
  6. 2024Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaborationcited by 4
  7. 2025Multimodal Representation Learning and Fusioncited by 3
  8. 2025MoRA: Missing Modality Low-Rank Adaptation for Visual Recognitioncited by 2
  9. 2026Prompting Particle Physics: Tokenized Multi-modal Foundation Models for Combinatorially Many Taskscited by 1
  10. 2024Sparsely Multimodal Data Fusioncited by 0
  11. 2025GSDNet: Revisiting Incomplete Multimodal-Diffusion from Graph Spectrum Perspective for Conversation Emotion Recognitioncited by 0
  12. 2026CALM: Interpretable Cross-Modal Alignment for Biomarker Discovery from Unpaired Datacited by 0