mlsanityData to 7 Oct 2026

Differentially private training of residual networks with scale normalisation

arXiv 2203.00324 ↗First posted 1 Mar 2022cs.LG cs.CRcited by 26

Not on the map. Why?

Abstract (arXiv)

The training of neural networks with Differentially Private Stochastic Gradient Descent offers formal Differential Privacy guarantees but introduces accuracy trade-offs. In this work, we propose to alleviate these trade-offs in residual networks with Group Normalisation through a simple architectural modification termed ScaleNorm by which an additional normalisation layer is introduced after the residual block's addition operation. Our method allows us to further improve on the recently reported state-of-the art on CIFAR-10, achieving a top-1 accuracy of 82.5% (ε=8.0) when trained from scratch.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 1 Mar 2022.

Builds on

20 of 33 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Layer Normalizationcited by 5,579
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  7. 2018Group Normalizationcited by 1,514
  8. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  9. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  10. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  11. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  12. 2018Scalable Private Learning with PATEcited by 266
8 more
  1. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  2. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 121
  3. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  4. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  5. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  6. 2021Not all noise is accounted equally: How differentially private learning benefits from large sampling ratescited by 21
  7. 2020On the effect of normalization layers on Differentially Private training of deep Neural networkscited by 6
  8. 2021Architecture Matters: Investigating the Influence of Differential Privacy on Neural Network Designcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 9; 2023: 10; 2024: 5; 2025: 2.

26 papers.

Sort:

  1. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  2. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  3. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  4. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  5. 2024How Private are DP-SGD Implementations?cited by 22
  6. 2022Differentially Private Learning with Per-Sample Adaptive Clippingcited by 21
  7. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  8. 2023Private, fair and accurate: Training large-scale, privacy-preserving AI models in medical imagingcited by 17
  9. 2023Exploring the Benefits of Visual Prompting in Differential Privacycited by 17
  10. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  11. 2022Differentially Private Image Classification from Featurescited by 7
  12. 2022Differentially Private Optimizers Can Learn Adversarially Robust Modelscited by 6
14 more
  1. 2023Preserving privacy in domain transfer of medical AI models comes at no performance costs: The integral role of differential privacycited by 6
  2. 2023Differential Privacy Meets Neural Network Pruningcited by 5
  3. 2023Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Modelscited by 4
  4. 2024Differential privacy enables fair and accurate AI-based analysis of speech disorders while protecting patient datacited by 4
  5. 2022SmoothNets: Optimizing CNN architecture design for differentially private deep learningcited by 3
  6. 2023Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classificationcited by 3
  7. 2022Bridging the Gap: Differentially Private Equivariant Deep Learning for Medical Image Analysiscited by 2
  8. 2023Reconciling AI Performance and Data Reconstruction Resilience for Medical Imagingcited by 1
  9. 2022Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learningcited by 0
  10. 2024Differentially Private Active Learning: Balancing Effective Data Selection and Privacycited by 0
  11. 2024An Architecture Built for Federated Learning: Addressing Data Heterogeneity through Adaptive Normalization-Free Feature Recalibrationcited by 0
  12. 2024Protecting Confidentiality, Privacy and Integrity in Collaborative Learningcited by 0
  13. 2025Differentially Private Relational Learning with Entity-level Privacy Guaranteescited by 0
  14. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0