mlsanityData to 7 Oct 2026

Reconstructing Training Data from Trained Neural Networks

arXiv 2206.07758 ↗First posted 15 Jun 2022cs.LG cs.CR cs.CVcited by 129

Not on the map. Why?

Abstract (arXiv)

Understanding to what extent neural networks memorize training data is an intriguing question with practical and theoretical implications. In this paper we show that in some cases a significant fraction of the training data can in fact be reconstructed from the parameters of a trained neural network classifier. We propose a novel reconstruction scheme that stems from recent theoretical results about the implicit bias in training neural networks with gradient-based methods. To the best of our knowledge, our results are the first to show that reconstructing a large portion of the actual training samples from a trained neural network classifier is generally possible. This has negative implications on privacy, as it can be used as an attack for revealing sensitive training data. We demonstrate our method for binary MLP classifiers on a few standard computer vision datasets.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 54 of its 85 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

54 of 85 references.

References · 54

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2019Deep Leakage from Gradientscited by 1,004
  9. 2018Robustness May Be at Odds with Accuracycited by 873
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  12. 2016Stealing Machine Learning Models via Prediction APIscited by 602
38 more
  1. 2017Exploring Generalization in Deep Learningcited by 595
  2. 2018Exploiting Unintended Feature Leakage in Collaborative Learningcited by 509
  3. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  4. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  5. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  7. 2014Understanding Deep Image Representations by Inverting Themcited by 399
  8. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  9. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  10. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  11. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  13. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  14. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  15. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  16. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  17. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 241
  18. 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 208
  19. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  20. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  21. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  22. 2020Directional convergence and alignment in deep learningcited by 172
  23. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  24. 2022Reconstructing Training Data with Informed Adversariescited by 143
  25. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  26. 2018Stealing Hyperparameters in Machine Learningcited by 115
  27. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  28. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  29. 2020InstaHide: Instance-hiding Schemes for Private Distributed Learningcited by 90
  30. 2018Model Reconstruction from Model Explanationscited by 84
  31. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 83
  32. 2019Reverse-Engineering Deep ReLU Networkscited by 81
  33. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  34. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  35. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  36. 2020Cryptanalytic Extraction of Neural Network Modelscited by 65
  37. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  38. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 39; 2024: 44; 2025: 25; 2026: 15.

129 papers.

Sort:

By area:

  1. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  3. 2023Can Neural Network Memorization Be Localized?cited by 66
  4. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  5. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  6. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  7. 2023Continual Learning: Applications and the Road Forwardcited by 59
  8. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  9. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  10. 2023Bounding Training Data Reconstruction in DP-SGDcited by 42
  11. 2023User Inference Attacks on Large Language Modelscited by 36
  12. 2023One-shot Empirical Privacy Estimation for Federated Learningcited by 35
38 more
  1. 2023Can Copyright be Reduced to Privacy?cited by 24
  2. 2023Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Lossescited by 24
  3. 2023The Double-Edged Sword of Implicit Bias: Generalization vs. Robustness in ReLU Networkscited by 23
  4. 2023Have it your way: Individualized Privacy Assignment for DP-SGDcited by 21
  5. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  6. 2023Gaussian Membership Inference Privacycited by 20
  7. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  8. 2023Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imagingcited by 17
  9. 2024Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Modelcited by 17
  10. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  11. 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
  12. 2023The Inadequacy of Similarity-based Privacy Metrics: Privacy Attacks against "Truly Anonymous" Synthetic Datasetscited by 14
  13. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  14. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  15. 2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
  16. 2024Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometricscited by 12
  17. 2024Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorizationcited by 12
  18. 2022Learn the Time to Learn: Replay Scheduling in Continual Learningcited by 11
  19. 2024Trained Random Forests Completely Reveal your Datasetcited by 11
  20. 2023Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right?cited by 10
  21. 2023Surrogate Model Extension (SME): A Fast and Accurate Weight Update Attack on Federated Learningcited by 10
  22. 2024Reconstructing Training Data From Real World Models Trained with Transfer Learningcited by 10
  23. 2023Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networkscited by 9
  24. 2024QUEEN: Query Unlearning against Model Extractioncited by 9
  25. 2023Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspacescited by 8
  26. 2023Collaborative Learning via Prediction Consensuscited by 8
  27. 2023Communication Efficient Private Federated Learning Using Ditheringcited by 8
  28. 2024Improving Robustness to Model Inversion Attacks via Sparse Coding Architecturescited by 8
  29. 2023Reconstructing Training Data from Multiclass Neural Networkscited by 7
  30. 2023One or Two Things We know about Concept Drift -- A Survey on Monitoring Evolving Environmentscited by 7
  31. 2024Efficiently Computing Similarities to Private Datasetscited by 7
  32. 2024Dataset Size Recovery from LoRA Weightscited by 7
  33. 2025Differential privacy for medical deep learning: methods, tradeoffs, and deployment implicationscited by 7
  34. 2023Understanding Deep Gradient Leakage via Inversion Influence Functionscited by 6
  35. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  36. 2023Learning Human Action Recognition Representations Without Real Humanscited by 5
  37. 2024Data Reconstruction Attacks and Defenses: A Systematic Evaluationcited by 5
  38. 2024Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm?cited by 5

50 most cited of 129 All 129 (JSON)