# Model Reconstruction from Model Explanations

arXiv: https://arxiv.org/abs/1807.05185 · HTML: https://mlsanity.com/paper/1807.05185/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2018.

## Built on by (most cited first, 30 of 86)

- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Reconstructing Training Data from Trained Neural Networks](https://mlsanity.com/paper/2206.07758/) (2022)
- [I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences](https://mlsanity.com/paper/2206.08451/) (2022)
- [Towards Data-Free Model Stealing in a Hard Label Setting](https://mlsanity.com/paper/2204.11022/) (2022)
- [Leveraging Explanations in Interactive Machine Learning: An Overview](https://mlsanity.com/paper/2207.14526/) (2022)
- [Exploiting Explanations for Model Inversion Attacks](https://mlsanity.com/paper/2104.12669/) (2021)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)
- [DeepSteal: Advanced Model Extractions Leveraging Efficient Weight Stealing in Memories](https://mlsanity.com/paper/2111.04625/) (2021)
- [Hidden symmetries of ReLU networks](https://mlsanity.com/paper/2306.06179/) (2023)
- [MEGEX: Data-Free Model Extraction Attack against Gradient-Based Explainable AI](https://mlsanity.com/paper/2107.08909/) (2021)
- [Gradient based Feature Attribution in Explainable AI: A Technical Review](https://mlsanity.com/paper/2403.10415/) (2024)
- [DualCF: Efficient Model Extraction Attack from Counterfactual Explanations](https://mlsanity.com/paper/2205.06504/) (2022)
- [Hardness of Noise-Free Learning for Two-Hidden-Layer Neural Networks](https://mlsanity.com/paper/2202.05258/) (2022)
- [Characterizing the risk of fairwashing](https://mlsanity.com/paper/2106.07504/) (2021)
- [Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud](https://mlsanity.com/paper/2109.03285/) (2021)
- [Stealing the Decoding Algorithms of Language Models](https://mlsanity.com/paper/2303.04729/) (2023)
- [Polynomial Time Cryptanalytic Extraction of Neural Network Models](https://mlsanity.com/paper/2310.08708/) (2023)
- [Inferring Sensitive Attributes from Model Explanations](https://mlsanity.com/paper/2208.09967/) (2022)
- [Model Reconstruction Using Counterfactual Explanations: A Perspective From Polytope Theory](https://mlsanity.com/paper/2405.05369/) (2024)
- [Differentially Private Counterfactuals via Functional Mechanism](https://mlsanity.com/paper/2208.02878/) (2022)
- [An Exact Poly-Time Membership-Queries Algorithm for Extraction a three-Layer ReLU Network](https://mlsanity.com/paper/2105.09673/) (2021)
- [Feature-based Learning for Diverse and Privacy-Preserving Counterfactual Explanations](https://mlsanity.com/paper/2209.13446/) (2022)
- [XRand: Differentially Private Defense against Explanation-Guided Attacks](https://mlsanity.com/paper/2212.04454/) (2022)
- [Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack using Public Data](https://mlsanity.com/paper/2302.08466/) (2023)
- [SoK: Machine Learning Governance](https://mlsanity.com/paper/2109.10870/) (2021)
- [Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks](https://mlsanity.com/paper/2308.00958/) (2023)
- [AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Interpretable Models](https://mlsanity.com/paper/2302.02162/) (2023)
- [A Survey on Explainable Artificial Intelligence for Cybersecurity](https://mlsanity.com/paper/2303.12942/) (2023)
- [Feature Inference Attack on Shapley Values](https://mlsanity.com/paper/2407.11359/) (2024)
- [Polynomial Time Cryptanalytic Extraction of Deep Neural Networks in the Hard-Label Setting](https://mlsanity.com/paper/2410.05750/) (2024)

## Abstract (arXiv)

We show through theory and experiment that gradient-based explanations of a model quickly reveal the model itself. Our results speak to a tension between the desire to keep a proprietary model secret and the ability to offer model explanations. On the theoretical side, we give an algorithm that provably learns a two-layer ReLU network in a setting where the algorithm may query the gradient of the model with respect to chosen inputs. The number of queries is independent of the dimension and nearly optimal in its dependence on the model size. Of interest not only from a learning-theoretic perspective, this result highlights the power of gradients rather than labels as a learning primitive. Complementing our theory, we give effective heuristics for reconstructing models from gradient explanations that are orders of magnitude more query-efficient than reconstruction attacks relying on prediction interfaces.