# An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?

arXiv: https://arxiv.org/abs/2011.14999 · HTML: https://mlsanity.com/paper/2011.14999/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2020.

## Built on by (most cited first, 27 of 27)

- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refitting](https://mlsanity.com/paper/2212.06803/) (2022)
- [Sensitivity-Aware Amortized Bayesian Inference](https://mlsanity.com/paper/2310.11122/) (2023)
- [Leveraging Expert Consistency to Improve Algorithmic Decision Support](https://mlsanity.com/paper/2101.09648/) (2021)
- [Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Box](https://mlsanity.com/paper/2304.05527/) (2023)
- [Group-Level Data Selection for Efficient Pretraining](https://mlsanity.com/paper/2502.14709/) (2025)
- [How Many and Which Training Points Would Need to be Removed to Flip this Prediction?](https://mlsanity.com/paper/2302.02169/) (2023)
- [Towards Bridging the Gaps between the Right to Explanation and the Right to be Forgotten](https://mlsanity.com/paper/2302.04288/) (2023)
- [What Data Enables Optimal Decisions? An Exact Characterization for Linear Optimization](https://mlsanity.com/paper/2505.21692/) (2025)
- [Toward a Taxonomy of Trust for Probabilistic Machine Learning](https://mlsanity.com/paper/2112.03270/) (2021)
- [Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings](https://mlsanity.com/paper/2508.11847/) (2025)
- [Robustness Auditing for Linear Regression: To Singularity and Beyond](https://mlsanity.com/paper/2410.07916/) (2024)
- [Rescaled Influence Functions: Accurate Data Attribution in High Dimension](https://mlsanity.com/paper/2506.06656/) (2025)
- [Relabeling Minimal Training Subset to Flip a Prediction](https://mlsanity.com/paper/2305.12809/) (2023)
- [Poisoning Bayesian Inference via Data Deletion and Replication](https://mlsanity.com/paper/2503.04480/) (2025)
- [The Approximate Fisher Influence Function: Faster Estimation of Data Influence in Statistical Models](https://mlsanity.com/paper/2407.08169/) (2024)
- [Testing Most Influential Sets](https://mlsanity.com/paper/2510.20372/) (2025)
- [A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation](https://mlsanity.com/paper/2605.15761/) (2026)
- [Statistical and Computational Guarantees for Influence Diagnostics](https://mlsanity.com/paper/2212.04014/) (2022)
- [Detecting clinician implicit biases in diagnoses using proximal causal inference](https://mlsanity.com/paper/2501.16399/) (2025)
- [Efficiently Verifiable Proofs of Data Attribution](https://mlsanity.com/paper/2508.10866/) (2025)
- [On the Accuracy of Newton Step and Influence Function Data Attributions](https://mlsanity.com/paper/2512.12572/) (2025)
- [Stress-Testing Causal Claims via Cardinality Repairs](https://mlsanity.com/paper/2512.02491/) (2025)
- [When unlearning is free: leveraging low influence points to reduce computational costs](https://mlsanity.com/paper/2512.05254/) (2025)
- [Minimum Specification Perturbation: Robustness as Distance-to-Falsification in Causal Inference](https://mlsanity.com/paper/2605.01579/) (2026)
- [Finding Most Influential Sets](https://mlsanity.com/paper/2606.05919/) (2026)
- [Influence Diagnostics in High-dimensional M-estimation: Precise Asymptotics](https://mlsanity.com/paper/2607.09250/) (2026)

## Abstract (arXiv)

Study samples often differ from the target populations of inference and policy decisions in non-random ways. Researchers typically believe that such departures from random sampling -- due to changes in the population over time and space, or difficulties in sampling truly randomly -- are small, and their corresponding impact on the inference should be small as well. We might therefore be concerned if the conclusions of our studies are excessively sensitive to a very small proportion of our sample data. We propose a method to assess the sensitivity of applied econometric conclusions to the removal of a small fraction of the sample. Manually checking the influence of all possible small subsets is computationally infeasible, so we use an approximation to find the most influential subset. Our metric, the "Approximate Maximum Influence Perturbation," is based on the classical influence function, and is automatically computable for common methods including (but not limited to) OLS, IV, MLE, GMM, and variational Bayes. We provide finite-sample error bounds on approximation performance. At minimal extra cost, we provide an exact finite-sample lower bound on sensitivity. We find that sensitivity is driven by a signal-to-noise ratio in the inference problem, is not reflected in standard errors, does not disappear asymptotically, and is not due to misspecification. While some empirical applications are robust, results of several influential economics papers can be overturned by removing less than 1% of the sample.