mlsanityData to 14 Oct 2026

Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling

arXiv 2307.14126 ↗First posted 26 Jul 2023cs.CVcited by 163

Not on the map. Why?

Abstract (arXiv)

The missing modality issue is critical but non-trivial to be solved by multi-modal models. Current methods aiming to handle the missing modality problem in multi-modal tasks, either deal with missing modalities only during evaluation or train separate models to handle specific missing modality settings. In addition, these models are designed for specific tasks, so for example, classification models are not easily adapted to segmentation tasks and vice versa. In this paper, we propose the Shared-Specific Feature Modelling (ShaSpec) method that is considerably simpler and more effective than competing approaches that address the issues above. ShaSpec is designed to take advantage of all available input modalities during training and evaluation by learning shared and specific features to better represent the input data. This is achieved from a strategy that relies on auxiliary tasks based on distribution alignment and domain classification, in addition to a residual feature fusion procedure. Also, the design simplicity of ShaSpec enables its easy adaptation to multiple tasks, such as classification and segmentation. Experiments are conducted on both medical image segmentation and computer vision classification, with results indicating that ShaSpec outperforms competing methods by a large margin. For instance, on BraTS2018, ShaSpec improves the SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole tumour. The code repository address is https://github.com/billhhh/ShaSpec/.

Key facts. Cited by 163 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

25 of 36 references.

References · 25

  1. 2022Generative Adversarial Networkscited by 5,627
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  4. 2018Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challengecited by 511
  5. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  6. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  7. 2021Localizing Visual Sounds the Hard Waycited by 170
  8. 2020Deep Multimodal Fusion by Channel Exchangingcited by 118
  9. 2016HeMIS: Hetero-Modal Image Segmentationcited by 103
  10. 2020Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertaintycited by 101
  11. 2022mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentationcited by 95
  12. 2021Trusted Multi-View Classificationcited by 94
13 more
  1. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  2. 2020Unpaired Multi-modal Segmentation via Knowledge Distillationcited by 81
  3. 2019Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentationcited by 68
  4. 2020Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusioncited by 68
  5. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  6. 2021Learning Disentangled Representations in the Imaging Domaincited by 46
  7. 2021ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalitiescited by 43
  8. 2015Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sourcescited by 42
  9. 2019Brain Tumor Segmentation on MRI with Missing Modalitiescited by 42
  10. 2020Soft Expert Reward Learning for Vision-and-Language Navigationcited by 37
  11. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  12. 2016Nesterov's Accelerated Gradient and Momentum as approximations to Regularised Update Descentcited by 26
  13. 2022Uncertainty-aware Multi-modal Learning via Cross-modal Random Network Predictioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 41; 2025: 51; 2026: 64.

163 papers.

Sort:

By area:

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  3. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  4. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 70
  5. 2023Single-Model and Any-Modality for Video Object Trackingcited by 65
  6. 2024DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistencycited by 53
  7. 2024Recent Advances of Multimodal Continual Learning: A Comprehensive Surveycited by 39
  8. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Responsecited by 36
  9. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
  10. 2025A Closer Look at Multimodal Representation Collapsecited by 28
  11. 2024Multi-modal Vision Pre-training for Medical Image Analysiscited by 23
  12. 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 23
38 more
  1. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  2. 2025Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 20
  3. 2024All in One Framework for Multimodal Re-identification in the Wildcited by 18
  4. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 18
  5. 2023Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrastcited by 17
  6. 2023Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarkscited by 17
  7. 2024Robust Multimodal Learning via Representation Decouplingcited by 17
  8. 2024PASSION: Towards Effective Incomplete Multi-Modal Medical Image Segmentation with Imbalanced Missing Ratescited by 17
  9. 2025Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learningcited by 15
  10. 2025Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncodercited by 13
  11. 2024Fourier Prompt Tuning for Modality-Incomplete Scene Segmentationcited by 11
  12. 2025Distilled Prompt Learning for Incomplete Multimodal Survival Predictioncited by 11
  13. 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
  14. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  15. 2025M4-SAR: A Multi-Resolution, Multi-Polarization, Multi-Scene, Multi-Source Dataset and Benchmark for optical-SAR Object Detectioncited by 10
  16. 2025MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Seriescited by 10
  17. 2024Radar and Camera Fusion for Object Detection and Tracking: A Comprehensive Surveycited by 9
  18. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  19. 2025ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Modelcited by 8
  20. 2025Quantum Federated Learning for Multimodal Data: A Modality-Agnostic Approachcited by 8
  21. 2025Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Datacited by 8
  22. 2025Decoupled Audio-Visual Dataset Distillationcited by 7
  23. 2024ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalitiescited by 6
  24. 2024Part-Whole Relational Fusion Towards Multi-Modal Scene Understandingcited by 6
  25. 2023What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?cited by 5
  26. 2024Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Datacited by 5
  27. 2024MOTIVE: A Drug-Target Interaction Graph For Inductive Link Predictioncited by 5
  28. 2024Towards the Causal Complete Cause of Multi-Modal Representation Learningcited by 5
  29. 2024Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotationscited by 5
  30. 2024MARIA: a Multimodal Transformer Model for Incomplete Healthcare Datacited by 5
  31. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 5
  32. 2023Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentationcited by 4
  33. 2024Can Text-to-image Model Assist Multi-modal Learning for Visual Recognition with Visual Modality Missing?cited by 4
  34. 2024Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaborationcited by 4
  35. 2024Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalizationcited by 4
  36. 2024Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensationcited by 4
  37. 2024Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modalitycited by 4
  38. 2024Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approachcited by 4

50 most cited of 163 All 163 (JSON)