mlsanityData to 7 Oct 2026

Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation

arXiv 2304.10756 ↗First posted 21 Apr 2023cs.CV cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Using multiple spatial modalities has been proven helpful in improving semantic segmentation performance. However, there are several real-world challenges that have yet to be addressed: (a) improving label efficiency and (b) enhancing robustness in realistic scenarios where modalities are missing at the test time. To address these challenges, we first propose a simple yet efficient multi-modal fusion mechanism Linear Fusion, that performs better than the state-of-the-art multi-modal models even with limited supervision. Second, we propose M3L: Multi-modal Teacher for Masked Modality Learning, a semi-supervised framework that not only improves the multi-modal performance but also makes the model robust to the realistic missing modality scenario using unlabeled data. We create the first benchmark for semi-supervised multi-modal semantic segmentation and also report the robustness to missing modalities. Our proposal shows an absolute improvement of up to 10% on robust mIoU above the most competitive baselines. Our code is available at https://github.com/harshm121/M3L

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 38 of its 53 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

38 of 53 references.

References · 38

  1. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  2. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  3. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 577
  4. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 463
  5. 2021Semi-Supervised Semantic Segmentation with Cross Pseudo Supervisioncited by 462
  6. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  7. 2020Semi-Supervised Semantic Segmentation with Cross-Consistency Trainingcited by 375
  8. 2021Unbiased Teacher for Semi-Supervised Object Detectioncited by 305
  9. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  10. 2022Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labelscited by 218
  11. 2020PseudoSeg: Designing Pseudo Labels for Semantic Segmentationcited by 180
  12. 2022Multimodal Token Fusion for Vision Transformerscited by 132
26 more
  1. 2020Guided Collaborative Training for Pixel-wise Semi-Supervised Learningcited by 120
  2. 2020Deep Multimodal Fusion by Channel Exchangingcited by 117
  3. 2021Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentationcited by 113
  4. 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 105
  5. 2021Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentationcited by 101
  6. 2019Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Networkcited by 98
  7. 2021ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentationcited by 75
  8. 2018Self-Supervised Model Adaptation for Multimodal Semantic Segmentationcited by 73
  9. 2022Unbiased Teacher v2: Semi-supervised Object Detection for Anchor-free and Anchor-based Detectorscited by 69
  10. 2019Hetero-Modal Variational Encoder-Decoder for Joint Modality Completion and Segmentationcited by 68
  11. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 65
  12. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 53
  13. 2022Semi-supervised Vision Transformers at Scalecited by 49
  14. 2019Brain Tumor Segmentation on MRI with Missing Modalitiescited by 42
  15. 2020Modality Dropout for Improved Performance-driven Talking Facescited by 32
  16. 2021Multiview Pseudo-Labeling for Semi-supervised Learning from Videocited by 32
  17. 2022SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalitiescited by 31
  18. 2019A Cross-Season Correspondence Dataset for Robust Semantic Segmentationcited by 28
  19. 2021Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusioncited by 25
  20. 2021Semi-Supervised Vision Transformerscited by 24
  21. 2019Robust Semantic Segmentation in Adverse Weather Conditions by means of Sensor Data Fusioncited by 18
  22. 2019UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradationcited by 17
  23. 2020Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speechcited by 17
  24. 2019A unified representation network for segmentation with missing modalitiescited by 15
  25. 2019RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentationcited by 12
  26. 2021A non-hierarchical attention network with modality dropout for textual response generation in multimodal dialogue systemscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 8; 2026: 8.

24 papers.

Sort:

  1. 2024Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalitiescited by 29
  2. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 27
  3. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  4. 2023SkyScenes: A Synthetic Dataset for Aerial Scene Understandingcited by 12
  5. 2024Fourier Prompt Tuning for Modality-Incomplete Scene Segmentationcited by 11
  6. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  7. 2024Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillationcited by 10
  8. 2025Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalancecited by 5
  9. 2024Learning Trimodal Relation for Audio-Visual Question Answering with Missing Modalitycited by 4
  10. 2025Robust Multimodal Learning via Cross-Modal Proxy Tokenscited by 2
  11. 2025Multimodal Online Federated Learning with Modality Missing in Internet of Thingscited by 2
  12. 2025CHARM: Collaborative Harmonization across Arbitrary Modalities for Modality-agnostic Semantic Segmentationcited by 2
12 more
  1. 2025Mitigating Modality Quantity and Quality Imbalance in Multimodal Online Federated Learningcited by 2
  2. 2025Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learningcited by 1
  3. 2026Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflectioncited by 1
  4. 2026SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Datacited by 1
  5. 2026GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depthcited by 1
  6. 2026Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projectioncited by 1
  7. 2025BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentationcited by 0
  8. 2025CLARGA: Multimodal Graph Representation Learning over Arbitrary Sets of Modalitiescited by 0
  9. 2026LARGO: Low-Rank Hypernetwork for Handling Missing Modalitiescited by 0
  10. 2026An Attention-based Model for Robust Forecasting with Missing Modalitycited by 0
  11. 2026Toward Reliable RGB-D Semantic Segmentation: Handling Missing Modalities via Condition Dropoutcited by 0
  12. 2026TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMscited by 0