mlsanityData to 14 Oct 2026

Fuse It or Lose It: Deep Fusion for Multimodal Simulation-Based Inference

arXiv 2311.10671 ↗First posted 17 Nov 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We present multimodal neural posterior estimation (MultiNPE), a method to integrate heterogeneous data from different sources in simulation-based inference with neural networks. Inspired by advances in deep fusion, it allows researchers to analyze data from different domains and infer the parameters of complex mathematical models with increased accuracy. We consider three fusion approaches for MultiNPE (early, late, hybrid) and evaluate their performance in three challenging experiments. MultiNPE not only outperforms single-source baselines on a reference task, but also achieves superior inference on scientific models from cognitive neuroscience and cardiology. We systematically investigate the impact of partially missing data on the different fusion strategies. Across our experiments, late and hybrid fusion techniques emerge as the methods of choice for practical applications of multimodal simulation-based inference.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 44 of its 90 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

44 of 90 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  5. 2016Density estimation using Real NVPcited by 2,042
  6. 2016Variational Inference: A Review for Statisticianscited by 1,472
  7. 2017Deep Setscited by 1,421
  8. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  9. 2019Neural Spline Flowscited by 596
  10. 2019The frontier of simulation-based inferencecited by 502
  11. 2022Transformers in Time Series: A Surveycited by 497
  12. 2022Multimodal Learning with Transformers: A Surveycited by 400
32 more
  1. 2023Multimodal Deep Learningcited by 258
  2. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
  3. 2018Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flowscited by 221
  4. 2019Automatic Posterior Transformation for Likelihood-Free Inferencecited by 219
  5. 2020BayesFlow: Learning complex stochastic models with invertible neural networkscited by 144
  6. 2018Validating Bayesian Inference Algorithms with Simulation-Based Calibrationcited by 124
  7. 2016Towards a Neural Statisticiancited by 121
  8. 2023Flow Matching for Scalable Simulation-Based Inferencecited by 101
  9. 2019Multimodal Intelligence: Representation Learning, Information Fusion, and Applicationscited by 89
  10. 2021Cross-Modality Fusion Transformer for Multispectral Object Detectioncited by 86
  11. 2020Actor-Transformers for Group Activity Recognitioncited by 84
  12. 2020Neural Approximate Sufficient Statistics for Implicit Modelscited by 74
  13. 2021Simulation Intelligence: Towards a New Generation of Scientific Methodscited by 68
  14. 2018Likelihood-free inference with emulator networkscited by 61
  15. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 61
  16. 2022Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Modelscited by 50
  17. 2021Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networkscited by 49
  18. 2022Compositional Score Modeling for Simulation-based Inferencecited by 48
  19. 2023Learning Robust Statistics for Simulation-based Inference under Model Misspecificationcited by 48
  20. 2022Variational methods for simulation-based inferencecited by 43
  21. 2023JANA: Jointly Amortized Neural Approximation of Complex Bayesian Modelscited by 41
  22. 2015Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Networkcited by 39
  23. 2018A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networkscited by 31
  24. 2019Learning Representations from Imperfect Time Series Data via Tensor Rank Regularizationcited by 30
  25. 2021Sequential Neural Posterior and Likelihood Approximationcited by 30
  26. 2022Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomycited by 16
  27. 2019Partially Exchangeable Networks and Architectures for Learning Summary Statistics in Approximate Bayesian Computationcited by 12
  28. 2016ABC random forests for Bayesian parameter inferencecited by 11
  29. 2023Simulation-based Inference for Cardiovascular Modelscited by 11
  30. 2019Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Seriescited by 10
  31. 2022Learning Summary Statistics for Bayesian Inference with Autoencoderscited by 9
  32. 2021Robust and integrative Bayesian neural networks for likelihood-free parameter inferencecited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024All-in-one simulation-based inferencecited by 74
  2. 2023Consistency Models for Scalable and Fast Simulation-Based Inferencecited by 22
  3. 2024Compositional simulation-based inference for time seriescited by 9