mlsanityData to 14 Oct 2026

Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses

arXiv 2307.01827 ↗First posted 4 Jul 2023cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Memorization of training data is an active research area, yet our understanding of the inner workings of neural networks is still in its infancy. Recently, Haim et al. (2022) proposed a scheme to reconstruct training samples from multilayer perceptron binary classifiers, effectively demonstrating that a large portion of training samples are encoded in the parameters of such networks. In this work, we extend their findings in several directions, including reconstruction from multiclass and convolutional neural networks. We derive a more general reconstruction scheme which is applicable to a wider range of loss functions such as regression losses. Moreover, we study the various factors that contribute to networks' susceptibility to such reconstruction schemes. Intriguingly, we observe that using weight decay during training increases reconstructability both in terms of quantity and quality. Additionally, we examine the influence of the number of neurons relative to the number of training samples on the reconstructability. Code: https://github.com/gonbuzaglo/decoreco

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 4 Jul 2023.

Builds on

33 of 50 references.

References · 33

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  6. 2019Deep Leakage from Gradientscited by 1,021
  7. 2017A Closer Look at Memorization in Deep Networkscited by 942
  8. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  10. 2023Extracting Training Data from Diffusion Modelscited by 732
  11. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  12. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
21 more
  1. 2023Erasing Concepts from Diffusion Modelscited by 539
  2. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  3. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  4. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  5. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  6. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  7. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 334
  8. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 317
  11. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  12. 2016Synthesizing the preferred inputs for neurons in neural networks via deep generator networkscited by 181
  13. 2020Directional convergence and alignment in deep learningcited by 178
  14. 2022Reconstructing Training Data with Informed Adversariescited by 148
  15. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  16. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 104
  17. 2022Fishing for User Data in Large-Batch Federated Learning via Gradient Magnificationcited by 73
  18. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 71
  19. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  20. 2022Reconstructing Training Data from Model Gradient, Provablycited by 27
  21. 2022Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentationscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 10; 2026: 7.

26 papers.

Sort:

By area:

  1. 2024Towards Adversarially Robust Dataset Distillation by Curvature Regularizationcited by 17
  2. 2024Reconstructing Training Data From Real World Models Trained with Transfer Learningcited by 12
  3. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 11
  4. 2025Towards Reliable Forgetting: A Survey on Machine Unlearning Verificationcited by 8
  5. 2024Risks When Sharing LoRA Fine-Tuned Diffusion Model Weightscited by 6
  6. 2024Training Data Reconstruction: Privacy due to Uncertainty?cited by 4
  7. 2025Approximating Language Model Training Data from Weightscited by 4
  8. 2024Evaluating of Machine Unlearning: Robustness Verification Without Prior Modificationscited by 3
  9. 2025A Law of Data Reconstruction for Random Features (and Beyond)cited by 3
  10. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  11. 2025ImpMIA: Leveraging Implicit Bias for Membership Inference Attackcited by 2
  12. 2023Reconciling AI Performance and Data Reconstruction Resilience for Medical Imagingcited by 1
14 more
  1. 2025No Prior, No Leakage: Revisiting Reconstruction Attacks in Trained Neural Networkscited by 1
  2. 2026Data Reconstruction: Identifiability and Optimization with Sample Splittingcited by 1
  3. 2024Data Quality in Edge Machine Learning: A State-of-the-Art Surveycited by 0
  4. 2024Slowing Down Forgetting in Continual Learningcited by 0
  5. 2024On the Reconstruction of Training Data from Group Invariant Networkscited by 0
  6. 2025Securing Transfer-Learned Networks with Reverse Homomorphic Encryptioncited by 0
  7. 2025Querying Kernel Methods Suffices for Reconstructing their Training Datacited by 0
  8. 2025Bayesian Perspective on Memorization and Reconstructioncited by 0
  9. 2025On Reconstructing Training Data From Bayesian Posteriors and Trained Modelscited by 0
  10. 2026Forgetting Similar Samples: Can Machine Unlearning Do it Better?cited by 0
  11. 2026Efficient Techniques for Data Reconstruction, with Finite-Width Recovery Guaranteescited by 0
  12. 2026Differential Privacy of Gaussian Process Posterior Samplingcited by 0
  13. 2026A Sharp Transition in Data Reconstruction under Differential Privacycited by 0
  14. 2026Discriminative Score Function: Turning Pretrained Models into Functional Generative Priorscited by 0