# Interactive Label Cleaning with Example-based Explanations

arXiv: https://arxiv.org/abs/2106.03922 · HTML: https://mlsanity.com/paper/2106.03922/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Limitations of the Empirical Fisher Approximation for Natural Gradient Descent](https://mlsanity.com/paper/1905.12558/) (2019)
- [Representer Point Selection for Explaining Deep Neural Networks](https://mlsanity.com/paper/1811.09720/) (2018)
- [On the Accuracy of Influence Functions for Measuring Group Effects](https://mlsanity.com/paper/1905.13289/) (2019)
- [Making deep neural networks right for the right scientific reasons by interacting with their explanations](https://mlsanity.com/paper/2001.05371/) (2020)
- [Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded](https://mlsanity.com/paper/1902.03751/) (2019)
- [FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging](https://mlsanity.com/paper/2012.15781/) (2020)
- [On Second-Order Group Influence Functions for Black-Box Predictions](https://mlsanity.com/paper/1911.00418/) (2019)
- [Interpreting Black Box Predictions using Fisher Kernels](https://mlsanity.com/paper/1810.10118/) (2018)
- [FIND: Human-in-the-Loop Debugging Deep Text Classifiers](https://mlsanity.com/paper/2010.04987/) (2020)
- [RelatIF: Identifying Explanatory Training Examples via Relative Influence](https://mlsanity.com/paper/2003.11630/) (2020)
- ["Influence Sketching": Finding Influential Samples In Large-Scale Regressions](https://mlsanity.com/paper/1611.05923/) (2016)
- [Machine Guides, Human Supervises: Interactive Learning with Global Explanations](https://mlsanity.com/paper/2009.09723/) (2020)
- [Learning in the Wild with Incremental Skeptical Gaussian Processes](https://mlsanity.com/paper/2011.00928/) (2020)

## Built on by (most cited first, 30 of 49)

- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Concept-level Debugging of Part-Prototype Networks](https://mlsanity.com/paper/2205.15769/) (2022)
- [Intriguing Properties of Data Attribution on Diffusion Models](https://mlsanity.com/paper/2311.00500/) (2023)
- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translation](https://mlsanity.com/paper/2405.12915/) (2024)
- [A Rationale-Centric Framework for Human-in-the-loop Machine Learning](https://mlsanity.com/paper/2203.12918/) (2022)
- [Reconstruction Attacks on Machine Unlearning: Simple Models are Vulnerable](https://mlsanity.com/paper/2405.20272/) (2024)
- [Understanding Influence Functions and Datamodels via Harmonic Analysis](https://mlsanity.com/paper/2210.01072/) (2022)
- [AQuA: A Benchmarking Tool for Label Quality Assessment](https://mlsanity.com/paper/2306.09467/) (2023)
- [FairIF: Boosting Fairness in Deep Learning via Influence Functions with Validation Set Sensitive Attributes](https://mlsanity.com/paper/2201.05759/) (2022)
- [How Many and Which Training Points Would Need to be Removed to Flip this Prediction?](https://mlsanity.com/paper/2302.02169/) (2023)
- [Better Training Data Attribution via Better Inverse Hessian-Vector Products](https://mlsanity.com/paper/2507.14740/) (2025)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [Error Discovery by Clustering Influence Embeddings](https://mlsanity.com/paper/2312.04712/) (2023)
- [May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability](https://mlsanity.com/paper/2309.13965/) (2023)
- [Toward a Unified Framework for Debugging Concept-based Models](https://mlsanity.com/paper/2109.11160/) (2021)
- [The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research](https://mlsanity.com/paper/2310.07882/) (2023)
- [Online Decision Mediation](https://mlsanity.com/paper/2310.18601/) (2023)
- [Relabeling Minimal Training Subset to Flip a Prediction](https://mlsanity.com/paper/2305.12809/) (2023)
- [Exploring Practitioner Perspectives On Training Data Attribution Explanations](https://mlsanity.com/paper/2310.20477/) (2023)
- [Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AI](https://mlsanity.com/paper/2409.16978/) (2024)
- [Influence Functions for Edge Edits in Non-Convex Graph Neural Networks](https://mlsanity.com/paper/2506.04694/) (2025)
- [LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution](https://mlsanity.com/paper/2601.21929/) (2026)
- [Efficient and Reliable Probabilistic Interactive Learning with Structured Outputs](https://mlsanity.com/paper/2202.08566/) (2022)
- [Lifelong Personal Context Recognition](https://mlsanity.com/paper/2205.10123/) (2022)
- [Influence-based Attributions can be Manipulated](https://mlsanity.com/paper/2409.05208/) (2024)
- [Detecting Instruction Fine-tuning Attacks using Influence Function](https://mlsanity.com/paper/2504.09026/) (2025)
- [A2LC: Active and Automated Label Correction for Semantic Segmentation](https://mlsanity.com/paper/2506.11599/) (2025)

## Abstract (arXiv)

We tackle sequential learning under label noise in applications where a human supervisor can be queried to relabel suspicious examples. Existing approaches are flawed, in that they only relabel incoming examples that look "suspicious" to the model. As a consequence, those mislabeled examples that elude (or don't undergo) this cleaning step end up tainting the training data and the model with no further chance of being cleaned. We propose Cincer, a novel approach that cleans both new and past data by identifying pairs of mutually incompatible examples. Whenever it detects a suspicious example, Cincer identifies a counter-example in the training set that -- according to the model -- is maximally incompatible with the suspicious example, and asks the annotator to relabel either or both examples, resolving this possible inconsistency. The counter-examples are chosen to be maximally incompatible, so to serve as explanations of the model's suspicion, and highly influential, so to convey as much information as possible if relabeled. Cincer achieves this by leveraging an efficient and robust approximation of influence functions based on the Fisher information matrix (FIM). Our extensive empirical evaluation shows that clarifying the reasons behind the model's suspicions by cleaning the counter-examples helps in acquiring substantially better data and models, especially when paired with our FIM approximation.