# Inferring Sensitive Attributes from Model Explanations

arXiv: https://arxiv.org/abs/2208.09967 · HTML: https://mlsanity.com/paper/2208.09967/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 21 Aug 2022.

## Builds on (most cited first, 26 of 26)

- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [On the (In)fidelity and Sensitivity for Explanations](https://mlsanity.com/paper/1901.09392/) (2019)
- [Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.02666/) (2021)
- [Fooling Neural Network Interpretations via Adversarial Model Manipulation](https://mlsanity.com/paper/1902.02041/) (2019)
- ["How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations](https://mlsanity.com/paper/1911.06473/) (2019)
- [Model Reconstruction from Model Explanations](https://mlsanity.com/paper/1807.05185/) (2018)
- [Overlearning Reveals Sensitive Attributes](https://mlsanity.com/paper/1905.11742/) (2019)
- [Interpretable Deep Learning under Fire](https://mlsanity.com/paper/1812.00891/) (2018)
- [AttriGuard: A Practical Defense Against Attribute Inference Attacks via Adversarial Machine Learning](https://mlsanity.com/paper/1805.04810/) (2018)
- [Leakage of Dataset Properties in Multi-Party Machine Learning](https://mlsanity.com/paper/2006.07267/) (2020)
- [Model extraction from counterfactual explanations](https://mlsanity.com/paper/2009.01884/) (2020)
- [Robust and Stable Black Box Explanations](https://mlsanity.com/paper/2011.06169/) (2020)
- [Measuring Data Leakage in Machine-Learning Models with Fisher Information](https://mlsanity.com/paper/2102.11673/) (2021)
- [Interpretable and Differentially Private Predictions](https://mlsanity.com/paper/1906.02004/) (2019)
- [Improved Adversarial Learning for Fair Classification](https://mlsanity.com/paper/1901.10443/) (2019)
- [Model Explanations with Differential Privacy](https://mlsanity.com/paper/2006.09129/) (2020)
- [Attribute Privacy: Framework and Mechanisms](https://mlsanity.com/paper/2009.04013/) (2020)
- [Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs](https://mlsanity.com/paper/2105.12049/) (2021)
- [Pufferfish Privacy Mechanisms for Correlated Data](https://mlsanity.com/paper/1603.03977/) (2016)
- [DYSAN: Dynamically sanitizing motion sensor data against sensitive inferences through adversarial networks](https://mlsanity.com/paper/2003.10325/) (2020)
- [Dikaios: Privacy Auditing of Algorithmic Fairness via Attribute Inference Attacks](https://mlsanity.com/paper/2202.02242/) (2022)
- [Feature Attributions and Counterfactual Explanations Can Be Manipulated](https://mlsanity.com/paper/2106.12563/) (2021)

## Built on by (most cited first, 15 of 15)

- [On the Alignment of Group Fairness with Attribute Privacy](https://mlsanity.com/paper/2211.10209/) (2022)
- [How Does a Deep Learning Model Architecture Impact Its Privacy? A Comprehensive Study of Privacy Attacks on CNNs and Transformers](https://mlsanity.com/paper/2210.11049/) (2022)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)
- [A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures](https://mlsanity.com/paper/2404.00673/) (2024)
- [Privacy-Preserving Algorithmic Recourse](https://mlsanity.com/paper/2311.14137/) (2023)
- [SoK: Taming the Triangle -- On the Interplays between Fairness, Interpretability and Privacy in Machine Learning](https://mlsanity.com/paper/2312.16191/) (2023)
- [Privacy Risks and Preservation Methods in Explainable Artificial Intelligence: A Scoping Review](https://mlsanity.com/paper/2505.02828/) (2025)
- [Towards a Game-theoretic Understanding of Explanation-based Membership Inference Attacks](https://mlsanity.com/paper/2404.07139/) (2024)
- [Towards integration of Privacy Enhancing Technologies in Explainable Artificial Intelligence](https://mlsanity.com/paper/2507.04528/) (2025)
- [Through Their Eyes: User Perceptions on Sensitive Attribute Inference of Social Media Videos by Visual Language Models](https://mlsanity.com/paper/2508.07658/) (2025)
- [DeepLeak: Privacy Enhancing Hardening of Model Explanations Against Membership Leakage](https://mlsanity.com/paper/2601.03429/) (2026)
- [Beyond XAI:Obstacles Towards Responsible AI](https://mlsanity.com/paper/2309.03638/) (2023)
- [TIER: Trajectory-Invariant Explanation Regularization for Membership Privacy](https://mlsanity.com/paper/2607.02903/) (2026)
- [SoK: Privacy Attacks on Machine Learning via Explainable AI](https://mlsanity.com/paper/2609.10627/) (2026)
- [QuanText: Protecting Dataset-Level Secrets in Textual Data Sharing](https://mlsanity.com/paper/2609.17995/) (2026)

## Abstract (arXiv)

Model explanations provide transparency into a trained machine learning model's blackbox behavior to a model builder. They indicate the influence of different input attributes to its corresponding model prediction. The dependency of explanations on input raises privacy concerns for sensitive user data. However, current literature has limited discussion on privacy risks of model explanations. We focus on the specific privacy risk of attribute inference attack wherein an adversary infers sensitive attributes of an input (e.g., race and sex) given its model explanations. We design the first attribute inference attack against model explanations in two threat models where model builder either (a) includes the sensitive attributes in training data and input or (b) censors the sensitive attributes by not including them in the training data and input. We evaluate our proposed attack on four benchmark datasets and four state-of-the-art algorithms. We show that an adversary can successfully infer the value of sensitive attributes from explanations in both the threat models accurately. Moreover, the attack is successful even by exploiting only the explanations corresponding to sensitive attributes. These suggest that our attack is effective against explanations and poses a practical threat to data privacy. On combining the model predictions (an attack surface exploited by prior attacks) with explanations, we note that the attack success does not improve. Additionally, the attack success on exploiting model explanations is better compared to exploiting only model predictions. These suggest that model explanations are a strong attack surface to exploit for an adversary.