mlsanityData to 14 Oct 2026

Conformal Fusion Under Missing Modalities

arXiv 2608.07183 ↗First posted 7 Aug 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Multimodal fusion architectures typically assume all modalities are available at inference, yet sensor failures, acquisition variability, and cost constraints routinely produce incomplete observations. Existing work treats modality absence as a prediction-accuracy problem, leaving a more basic question unanswered: whether a model's confidence estimates remain calibrated when an entire input stream is removed. We argue that missing-modality robustness and calibrated uncertainty are a single coupled property, and introduce Modality-Conditioned Conformal Fusion (MCCF), an architecture that addresses both at once. MCCF combines a multimodal bottleneck fusion backbone trained with modality dropout, per-modality evidential heads producing modality-decomposed Dirichlet distributions, and a Dempster-Shafer combination rule that fuses the per-modality evidence into a joint predictive distribution; an absent modality contributes vacuous evidence that is structurally ignored, so the fused uncertainty automatically reflects the reduced information without test-time imputation. A Mondrian conformal calibration module keyed on the modality-presence mask then provides finite-sample group-conditional coverage for every non-empty modality subset. MCCF is, to our knowledge, the first method with formal coverage guarantees under arbitrary modality availability through architectural integration rather than post-hoc recalibration, and the evidential decomposition yields per-modality vacuity scores that localise uncertainty to the absent modality responsible. Across a synthetic problem and three real multimodal benchmarks, MCCF holds its target coverage on every modality-presence subset, substantially narrows the coverage gap between full and partial modalities relative to a marginal split-conformal baseline, and imposes no measurable accuracy cost relative to temperature-scaled and evidential baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 65 references that are in the map. First posted on arXiv 7 Aug 2026.

Builds on

48 of 65 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2017Squeeze-and-Excitation Networkscited by 4,789
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  10. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  11. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
36 more
  1. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  5. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  6. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  7. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  8. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  9. 2019Conformal Prediction Under Covariate Shiftcited by 519
  10. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  11. 2022Multimodal Learning with Transformers: A Surveycited by 400
  12. 2020Classification with Valid and Adaptive Coveragecited by 396
  13. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  14. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  15. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
  16. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 366
  17. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
  18. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  19. 2019Deep Evidential Regressioncited by 355
  20. 2022Conformal prediction beyond exchangeabilitycited by 355
  21. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  22. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
  23. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 168
  24. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
  25. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
  26. 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 132
  27. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  28. 2021Learning Optimal Conformal Classifierscited by 112
  29. 2022mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentationcited by 95
  30. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
  31. 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
  32. 2022Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisationcited by 66
  33. 2022Training Uncertainty-Aware Classifiers with Conformalized Deep Learningcited by 64
  34. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
  35. 2023Conformal Prediction with Missing Valuescited by 25
  36. 2024Evidential time-to-event prediction with calibrated uncertainty quantificationcited by 7

Built on by

Not cited yet.