mlsanityData to 14 Oct 2026

Calibrating Multimodal Learning

arXiv 2306.01265 ↗First posted 2 Jun 2023cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Multimodal machine learning has achieved remarkable progress in a wide range of scenarios. However, the reliability of multimodal learning remains largely unexplored. In this paper, through extensive empirical studies, we identify current multimodal classification methods suffer from unreliable predictive confidence that tend to rely on partial modalities when estimating confidence. Specifically, we find that the confidence estimated by current models could even increase when some modalities are corrupted. To address the issue, we introduce an intuitive principle for multimodal learning, i.e., the confidence should not increase when one modality is removed. Accordingly, we propose a novel regularization technique, i.e., Calibrating Multimodal Learning (CML) regularization, to calibrate the predictive confidence of previous methods. This technique could be flexibly equipped by existing models and improve the performance in terms of confidence calibration, classification accuracy, and model robustness.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 48 of its 77 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

48 of 77 references.

References · 48

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  6. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  7. 2019When Does Label Smoothing Help?cited by 988
  8. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  9. 2014Distributed Representations of Sentences and Documentscited by 678
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  12. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
36 more
  1. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 344
  2. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  3. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  4. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  5. 2019Addressing Failure Prediction by Learning Model Confidencecited by 186
  6. 2020Training independent subnetworks for robust predictioncited by 177
  7. 2020Uncertainty Quantification and Deep Ensemblescited by 139
  8. 2021On Calibration and Out-of-domain Generalizationcited by 134
  9. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 129
  10. 2020Reliable Post hoc Explanations: Modeling Uncertainty in Explainabilitycited by 125
  11. 2020Confidence-Aware Learning for Deep Neural Networkscited by 113
  12. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  13. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 95
  14. 2021Soft Calibration Objectives for Neural Networkscited by 92
  15. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  16. 2019MFAS: Multimodal Fusion Architecture Searchcited by 84
  17. 2021Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classificationcited by 80
  18. 2021Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibrationcited by 79
  19. 2020Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantificationcited by 78
  20. 2020Neural Ensemble Search for Uncertainty Estimation and Dataset Shiftcited by 56
  21. 2020Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splittingcited by 54
  22. 2021SmoothMix: Training Confidence-calibrated Smoothed Classifiers for Certified Robustnesscited by 49
  23. 2021A Variational Information Bottleneck Approach to Multi-Omics Data Integrationcited by 40
  24. 2021On Model Calibration for Long-Tailed Object Detection and Instance Segmentationcited by 39
  25. 2021Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspectivecited by 39
  26. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  27. 2021Fairness in Ranking under Uncertaintycited by 27
  28. 2020Improving Calibration through the Relationship with Adversarial Robustnesscited by 26
  29. 2021BayesIMP: Uncertainty Quantification for Causal Data Fusioncited by 21
  30. 2021Disrupting Deep Uncertainty Estimation Without Harming Accuracycited by 19
  31. 2021A Geometric Perspective towards Neural Calibration via Sensitivity Decompositioncited by 19
  32. 2021Sparse Uncertainty Representation in Deep Learning with Inducing Weightscited by 16
  33. 2021Robustness via Uncertainty-aware Cycle Consistencycited by 16
  34. 2021Uncertainty Calibration for Ensemble-Based Debiasing Methodscited by 16
  35. 2020Hierarchical Gaussian Process Priors for Bayesian Neural Network Weightscited by 13
  36. 2021Understanding the Under-Coverage Bias in Uncertainty Estimationcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 7.

17 papers.

Sort:

By area:

  1. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 49
  2. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42
  3. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 40
  4. 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 23
  5. 2024Predictive Dynamic Fusioncited by 17
  6. 2024A Survey on Safe Multi-Modal Learning Systemcited by 15
  7. 2025BalanceBenchmark: A Survey for Multimodal Imbalance Learningcited by 8
  8. 2024Confidence-aware multi-modality learning for eye disease screeningcited by 5
  9. 2025Robust Multimodal Learning via Entropy-Gated Contrastive Fusioncited by 2
  10. 2026Knowing When to Answer: Adaptive Confidence Refinement for Reliable Audio-Visual Question Answeringcited by 2
  11. 2026CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusioncited by 2
  12. 2026BALM: A Model-Agnostic Framework for Balanced Multimodal Learning under Imbalanced Missing Ratescited by 1
5 more
  1. 2025Balanced Multimodal Learning: An Unidirectional Dynamic Interaction Perspectivecited by 0
  2. 2026Predicting Time Pressure of Powered Two-Wheeler Riders for Proactive Safety Interventionscited by 0
  3. 2026Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Modelscited by 0
  4. 2026Co-Learning for Missing Arbitrary Modalities in Multi-modal Classificationcited by 0
  5. 2026Boundary-Aligned Contribution Routing for Robust Optical--SAR Object Detectioncited by 0