# Practical multi-fidelity machine learning: fusion of deterministic and Bayesian models

arXiv: https://arxiv.org/abs/2407.15110 · HTML: https://mlsanity.com/paper/2407.15110/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 89 references that are in the map.
- First posted on arXiv 21 Jul 2024.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks](https://mlsanity.com/paper/1512.07666/) (2015)
- [Kernel methods through the roof: handling billions of points efficiently](https://mlsanity.com/paper/2006.10350/) (2020)
- [Transfer learning based multi-fidelity physics informed deep neural network](https://mlsanity.com/paper/2005.10614/) (2020)
- [Multi-fidelity Bayesian Neural Networks: Algorithms and Applications](https://mlsanity.com/paper/2012.13294/) (2020)
- [Deep Gaussian Processes for Multi-fidelity Modeling](https://mlsanity.com/paper/1903.07320/) (2019)
- [Multi-Fidelity Bayesian Optimization via Deep Neural Networks](https://mlsanity.com/paper/2007.03117/) (2020)
- [Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks](https://mlsanity.com/paper/2310.20447/) (2023)
- [Multi-fidelity regression using artificial neural networks: efficient approximation of parameter-dependent output quantities](https://mlsanity.com/paper/2102.13403/) (2021)
- [Latent Map Gaussian Processes for Mixed Variable Metamodeling](https://mlsanity.com/paper/2102.03935/) (2021)
- [Deep Learning: A Bayesian Perspective](https://mlsanity.com/paper/1706.00473/) (2017)
- [Multi-fidelity surrogate modeling using long short-term memory networks](https://mlsanity.com/paper/2208.03115/) (2022)
- [GP+: A Python Library for Kernel-based learning via Gaussian Processes](https://mlsanity.com/paper/2312.07694/) (2023)
- [Data Fusion with Latent Map Gaussian Processes](https://mlsanity.com/paper/2112.02206/) (2021)
- [Multi-fidelity Hierarchical Neural Processes](https://mlsanity.com/paper/2206.04872/) (2022)
- [A support vector regression-based multi-fidelity surrogate model](https://mlsanity.com/paper/1906.09439/) (2019)
- [A Latent Variable Approach for Non-Hierarchical Multi-Fidelity Adaptive Sampling](https://mlsanity.com/paper/2310.03298/) (2023)
- [Are you using test log-likelihood correctly?](https://mlsanity.com/paper/2212.00219/) (2022)
- [Fast and Accurate Gaussian Kernel Ridge Regression Using Matrix Decompositions for Preconditioning](https://mlsanity.com/paper/1905.10587/) (2019)
- [Transferring model structure in Bayesian transfer learning for Gaussian process regression](https://mlsanity.com/paper/2101.06884/) (2021)
- [Adaptivity for clustering-based reduced-order modeling of localized history-dependent phenomena](https://mlsanity.com/paper/2109.11897/) (2021)

## Built on by (most cited first, 4 of 4)

- [FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Models](https://mlsanity.com/paper/2601.22371/) (2026)
- [Surrogates for Physics-based and Data-driven Modelling of Parametric Systems: Review and New Perspectives](https://mlsanity.com/paper/2603.12870/) (2026)
- [Multifidelity Surrogate Modeling of Depressurized Loss of Forced Cooling in High-temperature Gas Reactors](https://mlsanity.com/paper/2603.14143/) (2026)
- [MAST: A Multi-fidelity Augmented Surrogate model via Spatial Trust-weighting](https://mlsanity.com/paper/2602.20974/) (2026)

## Abstract (arXiv)

Multi-fidelity machine learning methods address the accuracy-efficiency trade-off by integrating scarce, resource-intensive high-fidelity data with abundant but less accurate low-fidelity data. We propose a practical multi-fidelity strategy for problems spanning low- and high-dimensional domains, integrating a non-probabilistic regression model for the low-fidelity with a Bayesian model for the high-fidelity. The models are trained in a staggered scheme, where the low-fidelity model is transfer-learned to the high-fidelity data and a Bayesian model is trained to learn the residual between the data and the transfer-learned model. This three-model strategy -- deterministic low-fidelity, transfer-learning, and Bayesian residual -- leads to a prediction that includes uncertainty quantification for noisy and noiseless multi-fidelity data. The strategy is general and unifies the topic, highlighting the expressivity trade-off between the transfer-learning and Bayesian models (a complex transfer-learning model leads to a simpler Bayesian model, and vice versa). We propose modeling choices for two scenarios, and argue in favor of using a linear transfer-learning model that fuses 1) kernel ridge regression for low-fidelity with Gaussian processes for high-fidelity; or 2) deep neural network for low-fidelity with a Bayesian neural network for high-fidelity. We demonstrate the effectiveness and efficiency of the proposed strategies and contrast them with the state-of-the-art based on various numerical examples and two engineering problems. The results indicate that the proposed approach achieves comparable performance in both mean and uncertainty estimation while significantly reducing training time for machine learning modeling in data-scarce scenarios. Moreover, in data-rich settings, it outperforms other multi-fidelity architectures by effectively mitigating overfitting.