# How Many and Which Training Points Would Need to be Removed to Flip this Prediction?

arXiv: https://arxiv.org/abs/2302.02169 · HTML: https://mlsanity.com/paper/2302.02169/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 4 Feb 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [Input Similarity from the Neural Network Perspective](https://mlsanity.com/paper/2102.05262/) (2021)
- [Interactive Label Cleaning with Example-based Explanations](https://mlsanity.com/paper/2106.03922/) (2021)
- [Interpreting Black Box Predictions using Fisher Kernels](https://mlsanity.com/paper/1810.10118/) (2018)
- [Combining Feature and Instance Attribution to Detect Artifacts](https://mlsanity.com/paper/2107.00323/) (2021)
- [Can You Trust This Prediction? Auditing Pointwise Reliability After Learning](https://mlsanity.com/paper/1901.00403/) (2019)
- [RelatIF: Identifying Explanatory Training Examples via Relative Influence](https://mlsanity.com/paper/2003.11630/) (2020)
- [An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?](https://mlsanity.com/paper/2011.14999/) (2020)
- [Influence Tuning: Demoting Spurious Correlations via Instance Attribution and Instance-Driven Updates](https://mlsanity.com/paper/2110.03212/) (2021)
- [Optimal Sub-sampling with Influence Functions](https://mlsanity.com/paper/1709.01716/) (2017)
- [Disentangling Influence: Using Disentangled Representations to Audit Model Predictions](https://mlsanity.com/paper/1906.08652/) (2019)

## Built on by (most cited first, 10 of 10)

- [Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs](https://mlsanity.com/paper/2303.08114/) (2023)
- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [Unlearning Traces the Influential Training Data of Language Models](https://mlsanity.com/paper/2401.15241/) (2024)
- [A Bayesian Approach To Analysing Training Data Attribution In Deep Learning](https://mlsanity.com/paper/2305.19765/) (2023)
- [Better Training Data Attribution via Better Inverse Hessian-Vector Products](https://mlsanity.com/paper/2507.14740/) (2025)
- [Towards Practical Robustness Auditing for Linear Regression](https://mlsanity.com/paper/2307.16315/) (2023)
- [Relabeling Minimal Training Subset to Flip a Prediction](https://mlsanity.com/paper/2305.12809/) (2023)
- [Wrapper Boxes: Faithful Attribution of Model Predictions to Training Data](https://mlsanity.com/paper/2311.08644/) (2023)
- [How to Achieve Higher Accuracy with Less Training Points?](https://mlsanity.com/paper/2504.13586/) (2025)
- [On Robustness of Linear Classifiers to Targeted Data Poisoning](https://mlsanity.com/paper/2511.12722/) (2025)

## Abstract (arXiv)

We consider the problem of identifying a minimal subset of training data St such that if the instances comprising St had been removed prior to training, the categorization of a given test point xt would have been different. Identifying such a set may be of interest for a few reasons. First, the cardinality of St provides a measure of robustness (if |St| is small for xt, we might be less confident in the corresponding prediction), which we show is correlated with but complementary to predicted probabilities. Second, interrogation of St may provide a novel mechanism for contesting a particular model prediction: If one can make the case that the points in St are wrongly labeled or irrelevant, this may argue for overturning the associated prediction. Identifying St via brute-force is intractable. We propose comparatively fast approximation methods to find St based on influence functions, and find that -- for simple convex text classification models -- these approaches can often successfully identify relatively small sets of training examples which, if removed, would flip the prediction.