# Exploiting Explanations for Model Inversion Attacks

arXiv: https://arxiv.org/abs/2104.12669 · HTML: https://mlsanity.com/paper/2104.12669/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 57 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 30 of 33)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Not Just a Black Box: Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1605.01713/) (2016)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples](https://mlsanity.com/paper/1909.10594/) (2019)
- [A guide to convolution arithmetic for deep learning](https://mlsanity.com/paper/1603.07285/) (2016)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018)
- [Visual Interpretability for Deep Learning: a Survey](https://mlsanity.com/paper/1802.00614/) (2018)
- [Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning](https://mlsanity.com/paper/1904.01067/) (2019)
- [Model Reconstruction from Model Explanations](https://mlsanity.com/paper/1807.05185/) (2018)
- [Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations](https://mlsanity.com/paper/1904.02323/) (2019)
- [ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models](https://mlsanity.com/paper/1704.01942/) (2017)
- [Model extraction from counterfactual explanations](https://mlsanity.com/paper/2009.01884/) (2020)

## Built on by (most cited first, 30 of 49)

- [Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks](https://mlsanity.com/paper/2201.12179/) (2022)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [Re-thinking Model Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/2304.01669/) (2023)
- [Privacy Leakage on DNNs: A Survey of Model Inversion Attacks and Defenses](https://mlsanity.com/paper/2402.04013/) (2024)
- [SoK: Explainable Machine Learning for Computer Security Applications](https://mlsanity.com/paper/2208.10605/) (2022)
- [Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images](https://mlsanity.com/paper/2404.12922/) (2024)
- [On Strengthening and Defending Graph Reconstruction Attack with Markov Chain Approximation](https://mlsanity.com/paper/2306.09104/) (2023)
- [Model Reconstruction Using Counterfactual Explanations: A Perspective From Polytope Theory](https://mlsanity.com/paper/2405.05369/) (2024)
- [XRand: Differentially Private Defense against Explanation-Guided Attacks](https://mlsanity.com/paper/2212.04454/) (2022)
- [Model Inversion Attacks: A Survey of Approaches and Countermeasures](https://mlsanity.com/paper/2411.10023/) (2024)
- [On the utility and protection of optimization with differential privacy and classic regularization techniques](https://mlsanity.com/paper/2209.03175/) (2022)
- [Privacy Meets Explainability: A Comprehensive Impact Benchmark](https://mlsanity.com/paper/2211.04110/) (2022)
- [AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Interpretable Models](https://mlsanity.com/paper/2302.02162/) (2023)
- [Passive Inference Attacks on Split Learning via Adversarial Regularization](https://mlsanity.com/paper/2310.10483/) (2023)
- [Feature Inference Attack on Shapley Values](https://mlsanity.com/paper/2407.11359/) (2024)
- [Boosting Model Inversion Attacks with Adversarial Examples](https://mlsanity.com/paper/2306.13965/) (2023)
- [Improving Robustness to Model Inversion Attacks via Sparse Coding Architectures](https://mlsanity.com/paper/2403.14772/) (2024)
- [Model Inversion Attack via Dynamic Memory Learning](https://mlsanity.com/paper/2309.00013/) (2023)
- [SoK: Unintended Interactions among Machine Learning Defenses and Risks](https://mlsanity.com/paper/2312.04542/) (2023)
- [A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasures](https://mlsanity.com/paper/2404.00673/) (2024)
- [Theoretical Insights in Model Inversion Robustness and Conditional Entropy Maximization for Collaborative Inference Systems](https://mlsanity.com/paper/2503.00383/) (2025)
- [Privacy-Preserving Algorithmic Recourse](https://mlsanity.com/paper/2311.14137/) (2023)
- [Seven challenges for harmonizing explainability requirements](https://mlsanity.com/paper/2108.05390/) (2021)
- [Model Inversion Attack against Transfer Learning: Inverting a Model without Accessing It](https://mlsanity.com/paper/2203.06570/) (2022)
- [Generator Born from Classifier](https://mlsanity.com/paper/2312.02470/) (2023)
- [SoK: Taming the Triangle -- On the Interplays between Fairness, Interpretability and Privacy in Machine Learning](https://mlsanity.com/paper/2312.16191/) (2023)
- [UpCycling: Semi-supervised 3D Object Detection without Sharing Raw-level Unlabeled Scenes](https://mlsanity.com/paper/2211.11950/) (2022)
- [A Red Teaming Framework for Securing AI in Maritime Autonomous Systems](https://mlsanity.com/paper/2312.11500/) (2023)
- [Privacy Implications of Explainable AI in Data-Driven Systems](https://mlsanity.com/paper/2406.15789/) (2024)
- [A Survey on Privacy Attacks Against Digital Twin Systems in AI-Robotics](https://mlsanity.com/paper/2406.18812/) (2024)

## Abstract (arXiv)

The successful deployment of artificial intelligence (AI) in many domains from healthcare to hiring requires their responsible use, particularly in model explanations and privacy. Explainable artificial intelligence (XAI) provides more information to help users to understand model decisions, yet this additional knowledge exposes additional risks for privacy attacks. Hence, providing explanation harms privacy. We study this risk for image-based model inversion attacks and identified several attack architectures with increasing performance to reconstruct private image data from model explanations. We have developed several multi-modal transposed CNN architectures that achieve significantly higher inversion performance than using the target model prediction only. These XAI-aware inversion models were designed to exploit the spatial knowledge in image explanations. To understand which explanations have higher privacy risk, we analyzed how various explanation types and factors influence inversion performance. In spite of some models not providing explanations, we further demonstrate increased inversion performance even for non-explainable target models by exploiting explanations of surrogate models through attention transfer. This method first inverts an explanation from the target prediction, then reconstructs the target image. These threats highlight the urgent and significant privacy risks of explanations and calls attention for new privacy preservation techniques that balance the dual-requirement for AI explainability and privacy.