mlsanityData to 14 Oct 2026

Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation

arXiv 2304.01518 ↗First posted 4 Apr 2023cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Uncertainty estimation is an important research area to make deep neural networks (DNNs) more trustworthy. While extensive research on uncertainty estimation has been conducted with unimodal data, uncertainty estimation for multimodal data remains a challenge. Neural processes (NPs) have been demonstrated to be an effective uncertainty estimation method for unimodal data by providing the reliability of Gaussian processes with efficient and powerful DNNs. While NPs hold significant potential for multimodal uncertainty estimation, the adaptation of NPs for multimodal data has not been carefully studied. To bridge this gap, we propose Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. Based on the framework of NPs, MNPs consist of several novel and principled mechanisms tailored to the characteristics of multimodal data. In extensive empirical evaluation, our method achieves state-of-the-art multimodal uncertainty estimation performance, showing its appealing robustness against noisy samples and reliability in out-of-distribution detection with faster computation time compared to the current state-of-the-art multimodal uncertainty estimation method.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 46 of its 65 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

46 of 65 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2020Supervised Contrastive Learningcited by 2,829
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  9. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  11. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  12. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 519
34 more
  1. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  2. 2018Conditional Neural Processescited by 426
  3. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  4. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  5. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  6. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  7. 2018Neural Processescited by 311
  8. 2019Attentive Neural Processescited by 283
  9. 2014Scalable Variational Gaussian Process Classificationcited by 228
  10. 2015Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inferencecited by 159
  11. 2018Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentationcited by 146
  12. 2019Convolutional Conditional Neural Processescited by 142
  13. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  14. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  15. 2021Trusted Multi-View Classificationcited by 94
  16. 2021On Feature Collapse and Deep Kernel Learning for Single Forward Pass Uncertaintycited by 87
  17. 2020Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processescited by 76
  18. 2019Sequential Neural Processescited by 55
  19. 2018Dirichlet-based Gaussian Processes for Large-scale Calibrated Classificationcited by 51
  20. 2018Bayesian QuickNAT: Model Uncertainty in Deep Whole-Brain Segmentation for Structure-wise Quality Controlcited by 39
  21. 2021Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributionscited by 39
  22. 2019Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classificationcited by 36
  23. 2020Estimating Model Uncertainty of Neural Networks in Sparse Information Formcited by 31
  24. 2020Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processescited by 29
  25. 2022Latent Bottlenecked Attentive Neural Processescited by 26
  26. 2022NP-Match: When Neural Processes meet Semi-Supervised Learningcited by 25
  27. 2021Periodic Activation Functions Induce Stationaritycited by 21
  28. 2021Group Equivariant Conditional Neural Processescited by 16
  29. 2022Neural Processes with Stochastic Attention: Paying more attention to the context datasetcited by 16
  30. 2020Robustifying Sequential Neural Processescited by 15
  31. 2020Stationary Activations for Uncertainty Calibration in Deep Learningcited by 14
  32. 2022Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picturecited by 10
  33. 2020Message Passing Neural Processescited by 8
  34. 2021Evidential Turing Processescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 2.

11 papers.

  1. 2025Uncertainty Quantification for Machine Learning in Healthcare: A Surveycited by 14
  2. 2024Confidence-aware multi-modality learning for eye disease screeningcited by 5
  3. 2024LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Datacited by 5
  4. 2024Multimodal Learning with Uncertainty Quantification based on Discounted Belief Fusioncited by 5
  5. 2024Navigating Conflicting Views: Harnessing Trust for Learningcited by 4
  6. 2024R'enyi Neural Processescited by 1
  7. 2025Certainly Bot Or Not? Trustworthy Social Bot Detection via Robust Multi-Modal Neural Processescited by 1
  8. 2025Distributionally Robust Multimodal Machine Learningcited by 0
  9. 2025Supervised Learning of Random Neural Architectures Structured by Latent Random Fields on Compact Boundaryless Multiply-Connected Manifoldscited by 0
  10. 2026Geometry of Uncertainty: Learning Metric Spaces for Multimodal State Estimation in RLcited by 0
  11. 2026CRUISE: Vision-Language Model-Guided Uncertainty-Aware Cross-Modal Sensor Fusion for Robust Autonomous Drivingcited by 0