# FIRE: Multi-fidelity Regression with Distribution-conditioned In-context Learning using Tabular Foundation Models

arXiv: https://arxiv.org/abs/2601.22371 · HTML: https://mlsanity.com/paper/2601.22371/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 68 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Builds on (most cited first, 29 of 29)

- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO](https://mlsanity.com/paper/2109.06716/) (2021)
- [Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning](https://mlsanity.com/paper/1903.04703/) (2019)
- [Distribution Calibration for Regression](https://mlsanity.com/paper/1905.06023/) (2019)
- [Multi-Information Source Optimization](https://mlsanity.com/paper/1603.00389/) (2016)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [Deep Gaussian Processes for Multi-fidelity Modeling](https://mlsanity.com/paper/1903.07320/) (2019)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data](https://mlsanity.com/paper/2507.03971/) (2025)
- [Using stacking to average Bayesian predictive distributions](https://mlsanity.com/paper/1704.02030/) (2017)
- [DrivAerNet: A Parametric Car Dataset for Data-Driven Aerodynamic Design and Prediction](https://mlsanity.com/paper/2403.08055/) (2024)
- [In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization](https://mlsanity.com/paper/2404.16795/) (2024)
- [GP+: A Python Library for Kernel-based learning via Gaussian Processes](https://mlsanity.com/paper/2312.07694/) (2023)
- [BlendedNet: A Blended Wing Body Aircraft Dataset and Surrogate Model for Aerodynamic Predictions](https://mlsanity.com/paper/2509.07209/) (2025)
- [Multi-Fidelity Residual Neural Processes for Scalable Surrogate Modeling](https://mlsanity.com/paper/2402.18846/) (2024)
- [Residual Gaussian Process: A Tractable Nonparametric Bayesian Emulator for Multi-fidelity Simulations](https://mlsanity.com/paper/2104.03743/) (2021)
- [Multi-fidelity Gaussian process surrogate modeling for regression problems in physics](https://mlsanity.com/paper/2404.11965/) (2024)
- [Practical multi-fidelity machine learning: fusion of deterministic and Bayesian models](https://mlsanity.com/paper/2407.15110/) (2024)
- [GAR: Generalized Autoregression for Multi-Fidelity Fusion](https://mlsanity.com/paper/2301.05729/) (2023)
- [Multifidelity linear regression for scientific machine learning from scarce data](https://mlsanity.com/paper/2403.08627/) (2024)
- [Multifidelity Surrogate Models: A New Data Fusion Perspective](https://mlsanity.com/paper/2404.14456/) (2024)
- [Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning](https://mlsanity.com/paper/2510.21379/) (2025)
- [Warped multifidelity Gaussian processes for data fusion of skewed environmental data](https://mlsanity.com/paper/2407.20295/) (2024)

## Built on by (most cited first, 2 of 2)

- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [MAST: A Multi-fidelity Augmented Surrogate model via Spatial Trust-weighting](https://mlsanity.com/paper/2602.20974/) (2026)

## Abstract (arXiv)

Multi-fidelity (MF) regression often operates in regimes of extreme data imbalance, where the commonly-used Gaussian-process (GP) surrogates struggle with cubic scaling costs and overfit to sparse high-fidelity observations, limiting efficiency and generalization in real-world applications. We introduce FIRE, a training-free MF framework that couples tabular foundation models (TFMs) to perform zero-shot in-context Bayesian inference via a high-fidelity correction model conditioned on the low-fidelity model's posterior predictive distributions. This cross-fidelity information transfer via distributional summaries captures heteroscedastic errors, enabling robust residual learning without model retraining. Across 31 benchmark problems spanning synthetic and real-world tasks (e.g., DrivAerNet, LCBench), FIRE delivers a stronger performance-time trade-off than seven state-of-the-art GP-based or deep learning MF regression methods, ranking highest in accuracy and uncertainty quantification with runtime advantages. Limitations include context window constraints and dependence on the quality of the pre-trained TFM's.