# A Higher-Order Swiss Army Infinitesimal Jackknife

arXiv: https://arxiv.org/abs/1907.12116 · HTML: https://mlsanity.com/paper/1907.12116/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2019.

## Built on by (most cited first, 14 of 14)

- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refitting](https://mlsanity.com/paper/2212.06803/) (2022)
- [On the Trade-Off between Actionable Explanations and the Right to be Forgotten](https://mlsanity.com/paper/2208.14137/) (2022)
- [JAWS: Auditing Predictive Uncertainty Under Covariate Shift](https://mlsanity.com/paper/2207.10716/) (2022)
- [Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings](https://mlsanity.com/paper/2508.11847/) (2025)
- [Certified Data Removal Under High-dimensional Settings](https://mlsanity.com/paper/2505.07640/) (2025)
- [Approximate Data Deletion in Generative Models](https://mlsanity.com/paper/2206.14439/) (2022)
- [Rescaled Influence Functions: Accurate Data Attribution in High Dimension](https://mlsanity.com/paper/2506.06656/) (2025)
- [Approximate Leave-one-out Cross Validation for Regression with ℓ1 Regularizers (extended version)](https://mlsanity.com/paper/2310.17629/) (2023)
- [The Approximate Fisher Influence Function: Faster Estimation of Data Influence in Statistical Models](https://mlsanity.com/paper/2407.08169/) (2024)
- [Data-Driven Influence Functions for Optimization-Based Causal Inference](https://mlsanity.com/paper/2208.13701/) (2022)
- [Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence](https://mlsanity.com/paper/2507.20272/) (2025)
- [Orthogonal Bootstrap: Efficient Simulation of Input Uncertainty](https://mlsanity.com/paper/2404.19145/) (2024)
- [Estimating Population-Risk Curves Along Nonconvex Gradient Flows from the Training Sample](https://mlsanity.com/paper/2608.30261/) (2026)

## Abstract (arXiv)

Cross validation (CV) and the bootstrap are ubiquitous model-agnostic tools for assessing the error or variability of machine learning and statistical estimators. However, these methods require repeatedly re-fitting the model with different weighted versions of the original dataset, which can be prohibitively time-consuming. For sufficiently regular optimization problems the optimum depends smoothly on the data weights, and so the process of repeatedly re-fitting can be approximated with a Taylor series that can be often evaluated relatively quickly. The first-order approximation is known as the "infinitesimal jackknife" in the statistics literature and has been the subject of recent interest in machine learning for approximate CV. In this work, we consider high-order approximations, which we call the "higher-order infinitesimal jackknife" (HOIJ). Under mild regularity conditions, we provide a simple recursive procedure to compute approximations of all orders with finite-sample accuracy bounds. Additionally, we show that the HOIJ can be efficiently computed even in high dimensions using forward-mode automatic differentiation. We show that a linear approximation with bootstrap weights approximation is equivalent to those provided by asymptotic normal approximations. Consequently, the HOIJ opens up the possibility of enjoying higher-order accuracy properties of the bootstrap using local approximations. Consistency of the HOIJ for leave-one-out CV under different asymptotic regimes follows as corollaries from our finite-sample bounds under additional regularity assumptions. The generality of the computation and bounds motivate the name "higher-order Swiss Army infinitesimal jackknife."