# Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning

arXiv: https://arxiv.org/abs/2105.02410 · HTML: https://mlsanity.com/paper/2105.02410/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 14 of 14)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](https://mlsanity.com/paper/2004.13912/) (2020)
- [Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations](https://mlsanity.com/paper/1703.03717/) (2017)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Grad-CAM: Why did you say that?](https://mlsanity.com/paper/1611.07450/) (2016)
- [CXPlain: Causal Explanations for Model Interpretation under Uncertainty](https://mlsanity.com/paper/1910.12336/) (2019)
- [The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations](https://mlsanity.com/paper/1907.09294/) (2019)
- [Fairwashing: the risk of rationalization](https://mlsanity.com/paper/1901.09749/) (2019)
- [Interpretable Companions for Black-Box Models](https://mlsanity.com/paper/2002.03494/) (2020)
- [A Semiparametric Approach to Interpretable Machine Learning](https://mlsanity.com/paper/2006.04732/) (2020)

## Built on by (most cited first, 3 of 3)

- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Sparse Interaction Additive Networks via Feature Interaction Detection and Sparse Selection](https://mlsanity.com/paper/2209.09326/) (2022)
- [Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean](https://mlsanity.com/paper/2301.11862/) (2023)

## Abstract (arXiv)

We propose Partially Interpretable Estimators (PIE) which attribute a prediction to individual features via an interpretable model, while a (possibly) small part of the PIE prediction is attributed to the interaction of features via a black-box model, with the goal to boost the predictive performance while maintaining interpretability. As such, the interpretable model captures the main contributions of features, and the black-box model attempts to complement the interpretable piece by capturing the "nuances" of feature interactions as a refinement. We design an iterative training algorithm to jointly train the two types of models. Experimental results show that PIE is highly competitive to black-box models while outperforming interpretable baselines. In addition, the understandability of PIE is comparable to simple linear models as validated via a human evaluation.