# The Infinitesimal Jackknife and Combinations of Models

arXiv: https://arxiv.org/abs/2209.00147 · HTML: https://mlsanity.com/paper/2209.00147/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 31 Aug 2022.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife](https://mlsanity.com/paper/1311.4555/) (2013)
- [Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics](https://mlsanity.com/paper/1905.10651/) (2019)
- [Quantifying Epistemic Uncertainty in Deep Learning](https://mlsanity.com/paper/2110.12122/) (2021)
- [Boosting Random Forests to Reduce Bias; One-Step Boosted Forest and its Variance Estimate](https://mlsanity.com/paper/1803.08000/) (2018)
- [A Unified Framework for Random Forest Prediction Error Estimation](https://mlsanity.com/paper/1912.07435/) (2019)
- [Boulevard: Regularized Stochastic Gradient Boosted Trees and Their Limiting Distribution](https://mlsanity.com/paper/1806.09762/) (2018)
- [Generalised Boosted Forests](https://mlsanity.com/paper/2102.12561/) (2021)

## Built on by (most cited first, 5 of 5)

- [Post-hoc Probabilistic Vision-Language Models](https://mlsanity.com/paper/2412.06014/) (2024)
- [A Generic Approach for Reproducible Model Distillation](https://mlsanity.com/paper/2211.12631/) (2022)
- [Policy learning for many outcomes of interest: Combining optimal policy trees with multi-objective Bayesian optimisation](https://mlsanity.com/paper/2212.06312/) (2022)
- [Empirical Likelihood for Random Forests and Ensembles](https://mlsanity.com/paper/2511.13934/) (2025)
- [Statistical Inference for Explainable Boosting Machines](https://mlsanity.com/paper/2601.18857/) (2026)

## Abstract (arXiv)

The Infinitesimal Jackknife is a general method for estimating variances of parametric models, and more recently also for some ensemble methods. In this paper we extend the Infinitesimal Jackknife to estimate the covariance between any two models. This can be used to quantify uncertainty for combinations of models, or to construct test statistics for comparing different models or ensembles of models fitted using the same training dataset. Specific examples in this paper use boosted combinations of models like random forests and M-estimators. We also investigate its application on neural networks and ensembles of XGBoost models. We illustrate the efficacy of variance estimates through extensive simulations and its application to the Beijing Housing data, and demonstrate the theoretical consistency of the Infinitesimal Jackknife covariance estimate.