# On the Privacy Risks of Model Explanations

arXiv: https://arxiv.org/abs/1907.00164 · HTML: https://mlsanity.com/paper/1907.00164/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2019.

## Built on by (most cited first, 7 of 7)

- [Exploiting Explanations for Model Inversion Attacks](https://mlsanity.com/paper/2104.12669/) (2021)
- [Are Your Sensitive Attributes Private? Novel Model Inversion Attribute Inference Attacks on Classification Models](https://mlsanity.com/paper/2201.09370/) (2022)
- [A Survey on Heterogeneous Federated Learning](https://mlsanity.com/paper/2210.04505/) (2022)
- [The privacy issue of counterfactual explanations: explanation linkage attacks](https://mlsanity.com/paper/2210.12051/) (2022)
- [A Survey on Explainable Artificial Intelligence for Cybersecurity](https://mlsanity.com/paper/2303.12942/) (2023)
- [Model Extraction and Defenses on Generative Adversarial Networks](https://mlsanity.com/paper/2101.02069/) (2021)
- [A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures](https://mlsanity.com/paper/2404.00673/) (2024)

## Abstract (arXiv)

Privacy and transparency are two key foundations of trustworthy machine learning. Model explanations offer insights into a model's decisions on input data, whereas privacy is primarily concerned with protecting information about the training data. We analyze connections between model explanations and the leakage of sensitive information about the model's training set. We investigate the privacy risks of feature-based model explanations using membership inference attacks: quantifying how much model predictions plus their explanations leak information about the presence of a datapoint in the training set of a model. We extensively evaluate membership inference attacks based on feature-based model explanations, over a variety of datasets. We show that backpropagation-based explanations can leak a significant amount of information about individual training datapoints. This is because they reveal statistical information about the decision boundaries of the model about an input, which can reveal its membership. We also empirically investigate the trade-off between privacy and explanation quality, by studying the perturbation-based model explanations.