# Training Set Debugging Using Trusted Items

arXiv: https://arxiv.org/abs/1801.08019 · HTML: https://mlsanity.com/paper/1801.08019/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jan 2018.

## Built on by (most cited first, 26 of 26)

- [De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks](https://mlsanity.com/paper/2105.03592/) (2021)
- [Interpretable Data-Based Explanations for Fairness Debugging](https://mlsanity.com/paper/2112.09745/) (2021)
- [Defense Against Reward Poisoning Attacks in Reinforcement Learning](https://mlsanity.com/paper/2102.05776/) (2021)
- [The Dataset Multiplicity Problem: How Unreliable Data Impacts Predictions](https://mlsanity.com/paper/2304.10655/) (2023)
- [Poisoning Attacks and Defenses on Artificial Intelligence: A Survey](https://mlsanity.com/paper/2202.10276/) (2022)
- [Iterative Teaching by Label Synthesis](https://mlsanity.com/paper/2110.14432/) (2021)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective](https://mlsanity.com/paper/2401.04374/) (2024)
- [Regularization Can Help Mitigate Poisoning Attacks... with the Right Hyperparameters](https://mlsanity.com/paper/2105.10948/) (2021)
- [Iterative Teaching by Data Hallucination](https://mlsanity.com/paper/2210.17467/) (2022)
- [Consistent Range Approximation for Fair Predictive Modeling](https://mlsanity.com/paper/2212.10839/) (2022)
- [Non-Clashing Teaching Maps for Balls in Graphs](https://mlsanity.com/paper/2309.02876/) (2023)
- [Certifying Data-Bias Robustness in Linear Regression](https://mlsanity.com/paper/2206.03575/) (2022)
- [Hyperparameter Learning under Data Poisoning: Analysis of the Influence of Regularization via Multiobjective Bilevel Optimization](https://mlsanity.com/paper/2306.01613/) (2023)
- [The Computational Complexity of Positive Non-Clashing Teaching in Graphs](https://mlsanity.com/paper/2503.07665/) (2025)
- [CHEF: A Cheap and Fast Pipeline for Iteratively Cleaning Label Uncertainties (Technical Report)](https://mlsanity.com/paper/2107.08588/) (2021)
- [New data poison attacks on machine learning classifiers for mobile exfiltration](https://mlsanity.com/paper/2210.11592/) (2022)
- [Defuse: Harnessing Unrestricted Adversarial Examples for Debugging Models Beyond Test Accuracy](https://mlsanity.com/paper/2102.06162/) (2021)
- [DREAM: Debugging and Repairing AutoML Pipelines](https://mlsanity.com/paper/2401.00379/) (2023)
- [Non-Clashing Teaching in Graphs: Algorithms, Complexity, and Bounds](https://mlsanity.com/paper/2602.00657/) (2026)
- [Manipulating Predictions over Discrete Inputs in Machine Teaching](https://mlsanity.com/paper/2401.17865/) (2024)
- [Learning from Uncertain Data: From Possible Worlds to Possible Models](https://mlsanity.com/paper/2405.18549/) (2024)
- [A Comparative Analysis of Influence Signals for Data Debugging](https://mlsanity.com/paper/2506.11584/) (2025)
- [Stealthy Poisoning Attacks Bypass Defenses in Regression Settings](https://mlsanity.com/paper/2601.22308/) (2026)
- [SAGE: Similarity-Based Cleaning of Poisoned Training Data from Verified Examples](https://mlsanity.com/paper/2610.01788/) (2026)
- [WisdomNet: trustable machine learning toward error-free classification](https://mlsanity.com/paper/2610.15299/) (2026)

## Abstract (arXiv)

Training set bugs are flaws in the data that adversely affect machine learning. The training set is usually too large for man- ual inspection, but one may have the resources to verify a few trusted items. The set of trusted items may not by itself be adequate for learning, so we propose an algorithm that uses these items to identify bugs in the training set and thus im- proves learning. Specifically, our approach seeks the smallest set of changes to the training set labels such that the model learned from this corrected training set predicts labels of the trusted items correctly. We flag the items whose labels are changed as potential bugs, whose labels can be checked for veracity by human experts. To find the bugs in this way is a challenging combinatorial bilevel optimization problem, but it can be relaxed into a continuous optimization problem. Ex- periments on toy and real data demonstrate that our approach can identify training set bugs effectively and suggest appro- priate changes to the labels. Our algorithm is a step toward trustworthy machine learning.