Conformal Fusion Under Missing Modalities
Not on the map. Why?
Abstract (arXiv)
Multimodal fusion architectures typically assume all modalities are available at inference, yet sensor failures, acquisition variability, and cost constraints routinely produce incomplete observations. Existing work treats modality absence as a prediction-accuracy problem, leaving a more basic question unanswered: whether a model's confidence estimates remain calibrated when an entire input stream is removed. We argue that missing-modality robustness and calibrated uncertainty are a single coupled property, and introduce Modality-Conditioned Conformal Fusion (MCCF), an architecture that addresses both at once. MCCF combines a multimodal bottleneck fusion backbone trained with modality dropout, per-modality evidential heads producing modality-decomposed Dirichlet distributions, and a Dempster-Shafer combination rule that fuses the per-modality evidence into a joint predictive distribution; an absent modality contributes vacuous evidence that is structurally ignored, so the fused uncertainty automatically reflects the reduced information without test-time imputation. A Mondrian conformal calibration module keyed on the modality-presence mask then provides finite-sample group-conditional coverage for every non-empty modality subset. MCCF is, to our knowledge, the first method with formal coverage guarantees under arbitrary modality availability through architectural integration rather than post-hoc recalibration, and the evidential decomposition yields per-modality vacuity scores that localise uncertainty to the absent modality responsible. Across a synthetic problem and three real multimodal benchmarks, MCCF holds its target coverage on every modality-presence subset, substantially narrows the coverage gap between full and partial modalities relative to a marginal split-conformal baseline, and imposes no measurable accuracy cost relative to temperature-scaled and evidential baselines.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 65 references that are in the map. First posted on arXiv 7 Aug 2026.
Builds on
48 of 65 references.
References · 48
- 2017Attention Is All You Needcited by 60,361
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2017Squeeze-and-Excitation Networkscited by 4,789
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2018CBAM: Convolutional Block Attention Modulecited by 2,477
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
36 more
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
- 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
- 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
- 2016Distribution-Free Predictive Inference For Regressioncited by 668
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
- 2019Conformal Prediction Under Covariate Shiftcited by 519
- 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
- 2022Multimodal Learning with Transformers: A Surveycited by 400
- 2020Classification with Valid and Adaptive Coveragecited by 396
- 2021Revisiting the Calibration of Modern Neural Networkscited by 390
- 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
- 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
- 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 366
- 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 365
- 2021Attention Bottlenecks for Multimodal Fusioncited by 361
- 2019Deep Evidential Regressioncited by 355
- 2022Conformal prediction beyond exchangeabilitycited by 355
- 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
- 2022Are Multimodal Transformers Robust to Missing Modality?cited by 186
- 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 168
- 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 163
- 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 146
- 2019UR-FUNNY: A Multimodal Language Dataset for Understanding Humorcited by 132
- 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
- 2021Learning Optimal Conformal Classifierscited by 112
- 2022mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentationcited by 95
- 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 92
- 2020Bayesian Deep Learning via Subnetwork Inferencecited by 87
- 2022Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisationcited by 66
- 2022Training Uncertainty-Aware Classifiers with Conformalized Deep Learningcited by 64
- 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 29
- 2023Conformal Prediction with Missing Valuescited by 25
- 2024Evidential time-to-event prediction with calibrated uncertainty quantificationcited by 7
Built on by
Not cited yet.