# What made you do this? Understanding black-box decisions with sufficient input subsets

arXiv: https://arxiv.org/abs/1810.03805 · HTML: https://mlsanity.com/paper/1810.03805/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2018.

## Built on by (most cited first, 30 of 31)

- [Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy](https://mlsanity.com/paper/2103.03014/) (2021)
- [Optimizing Relevance Maps of Vision Transformers Improves Robustness](https://mlsanity.com/paper/2206.01161/) (2022)
- [Zorro: Valid, Sparse, and Stable Explanations in Graph Neural Networks](https://mlsanity.com/paper/2105.08621/) (2021)
- [LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity](https://mlsanity.com/paper/2404.03214/) (2024)
- [Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior](https://mlsanity.com/paper/2107.09234/) (2021)
- [Local vs. Global Interpretability: A Computational Complexity Perspective](https://mlsanity.com/paper/2406.02981/) (2024)
- [Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features](https://mlsanity.com/paper/2111.03740/) (2021)
- [Explaining, Fast and Slow: Abstraction and Refinement of Provable Explanations](https://mlsanity.com/paper/2506.08505/) (2025)
- [Comparing the Decision-Making Mechanisms by Transformers and CNNs via Explanation Methods](https://mlsanity.com/paper/2212.06872/) (2022)
- [Explain Yourself, Briefly! Self-Explaining Neural Networks with Concise Sufficient Reasons](https://mlsanity.com/paper/2502.03391/) (2025)
- [Saliency Cards: A Framework to Characterize and Compare Saliency Methods](https://mlsanity.com/paper/2206.02958/) (2022)
- [New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound](https://mlsanity.com/paper/2211.02912/) (2022)
- [Task Discovery: Finding the Tasks that Neural Networks Generalize on](https://mlsanity.com/paper/2212.00261/) (2022)
- [Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships](https://mlsanity.com/paper/2407.12543/) (2024)
- [Self-Explaining Neural Networks for Business Process Monitoring](https://mlsanity.com/paper/2503.18067/) (2025)
- [Survey on AI Ethics: A Socio-technical Perspective](https://mlsanity.com/paper/2311.17228/) (2023)
- [An Additive Instance-Wise Approach to Multi-class Model Interpretation](https://mlsanity.com/paper/2207.03113/) (2022)
- [Additive Models Explained: A Computational Complexity Approach](https://mlsanity.com/paper/2510.21292/) (2025)
- [When Stability meets Sufficiency: Informative Explanations that do not Overwhelm](https://mlsanity.com/paper/2109.06181/) (2021)
- [Monetizing Currency Pair Sentiments through LLM Explainability](https://mlsanity.com/paper/2407.19922/) (2024)
- [Unifying Formal Explanations: A Complexity-Theoretic Perspective](https://mlsanity.com/paper/2602.18160/) (2026)
- [Explaining Classifiers Trained on Raw Hierarchical Multiple-Instance Data](https://mlsanity.com/paper/2208.02694/) (2022)
- [Provably Explaining Neural Additive Models](https://mlsanity.com/paper/2602.17530/) (2026)
- [Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy](https://mlsanity.com/paper/2402.01095/) (2024)
- [Succinct Interaction-Aware Explanations](https://mlsanity.com/paper/2402.05566/) (2024)
- [DD-CAM: Minimal Sufficient Explanations for Vision Models Using Delta Debugging](https://mlsanity.com/paper/2602.19274/) (2026)
- [Multimodal Data Curation Through Ranked Retrieval](https://mlsanity.com/paper/2605.01163/) (2026)
- [Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understanding](https://mlsanity.com/paper/2606.25160/) (2026)
- [What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks](https://mlsanity.com/paper/2607.09164/) (2026)
- [Counterfactual Fragility Certificates: Exposing High-Confidence Brittleness under Structured Evidence Failure](https://mlsanity.com/paper/2609.00366/) (2026)

## Abstract (arXiv)

Local explanation frameworks aim to rationalize particular decisions made by a black-box prediction model. Existing techniques are often restricted to a specific type of predictor or based on input saliency, which may be undesirably sensitive to factors unrelated to the model's decision making process. We instead propose sufficient input subsets that identify minimal subsets of features whose observed values alone suffice for the same decision to be reached, even if all other input feature values are missing. General principles that globally govern a model's decision-making can also be revealed by searching for clusters of such input patterns across many data points. Our approach is conceptually straightforward, entirely model-agnostic, simply implemented using instance-wise backward selection, and able to produce more concise rationales than existing techniques. We demonstrate the utility of our interpretation method on various neural network models trained on text, image, and genomic data.