# Interpreting Black Box Predictions using Fisher Kernels

arXiv: https://arxiv.org/abs/1810.10118 · HTML: https://mlsanity.com/paper/1810.10118/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2018.

## Built on by (most cited first, 30 of 48)

- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Training Data Influence Analysis and Estimation: A Survey](https://mlsanity.com/paper/2212.04612/) (2022)
- [Explanation-Based Human Debugging of NLP Models: A Survey](https://mlsanity.com/paper/2104.15135/) (2021)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Interactive Label Cleaning with Example-based Explanations](https://mlsanity.com/paper/2106.03922/) (2021)
- [Intriguing Properties of Data Attribution on Diffusion Models](https://mlsanity.com/paper/2311.00500/) (2023)
- [Evaluating Data Attribution for Text-to-Image Models](https://mlsanity.com/paper/2306.09345/) (2023)
- [The Journey, Not the Destination: How Data Guides Diffusion Models](https://mlsanity.com/paper/2312.06205/) (2023)
- [HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks](https://mlsanity.com/paper/2102.02515/) (2021)
- [DISSECT: Disentangled Simultaneous Explanations via Concept Traversals](https://mlsanity.com/paper/2105.15164/) (2021)
- [First is Better Than Last for Language Data Influence](https://mlsanity.com/paper/2202.11844/) (2022)
- [Training Data Attribution via Approximate Unrolled Differentiation](https://mlsanity.com/paper/2405.12186/) (2024)
- [Most Influential Subset Selection: Challenges, Promises, and Beyond](https://mlsanity.com/paper/2409.18153/) (2024)
- [Repairing Neural Networks by Leaving the Right Past Behind](https://mlsanity.com/paper/2207.04806/) (2022)
- [Human-Centered Concept Explanations for Neural Networks](https://mlsanity.com/paper/2202.12451/) (2022)
- [Proto2Proto: Can you recognize the car, the way I do?](https://mlsanity.com/paper/2204.11830/) (2022)
- [Understanding Influence Functions and Datamodels via Harmonic Analysis](https://mlsanity.com/paper/2210.01072/) (2022)
- [Diffusion Attribution Score: Evaluating Training Data Influence in Diffusion Models](https://mlsanity.com/paper/2410.18639/) (2024)
- [Understanding Instance-based Interpretability of Variational Auto-Encoders](https://mlsanity.com/paper/2105.14203/) (2021)
- [Concept Gradient: Concept-based Interpretation Without Linear Assumption](https://mlsanity.com/paper/2208.14966/) (2022)
- [Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation](https://mlsanity.com/paper/2401.09031/) (2024)
- [Natural Example-Based Explainability: a Survey](https://mlsanity.com/paper/2309.03234/) (2023)
- [Influence Estimation for Generative Adversarial Networks](https://mlsanity.com/paper/2101.08367/) (2021)
- [A Bayesian Approach To Analysing Training Data Attribution In Deep Learning](https://mlsanity.com/paper/2305.19765/) (2023)
- [How Many and Which Training Points Would Need to be Removed to Flip this Prediction?](https://mlsanity.com/paper/2302.02169/) (2023)
- [RestoreX-AI: A Contrastive Approach towards Guiding Image Restoration via Explainable AI Systems](https://mlsanity.com/paper/2204.01719/) (2022)
- [Influence Functions for Sequence Tagging Models](https://mlsanity.com/paper/2210.14177/) (2022)
- [First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation](https://mlsanity.com/paper/2511.04715/) (2025)
- [DIVINE: Diverse Influential Training Points for Data Visualization and Model Refinement](https://mlsanity.com/paper/2107.05978/) (2021)

## Abstract (arXiv)

Research in both machine learning and psychology suggests that salient examples can help humans to interpret learning models. To this end, we take a novel look at black box interpretation of test predictions in terms of training examples. Our goal is to ask `which training examples are most responsible for a given set of predictions'? To answer this question, we make use of Fisher kernels as the defining feature embedding of each data point, combined with Sequential Bayesian Quadrature (SBQ) for efficient selection of examples. In contrast to prior work, our method is able to seamlessly handle any sized subset of test predictions in a principled way. We theoretically analyze our approach, providing novel convergence bounds for SBQ over discrete candidate atoms. Our approach recovers the application of influence functions for interpretability as a special case yielding novel insights from this connection. We also present applications of the proposed approach to three use cases: cleaning training data, fixing mislabeled examples and data summarization.