mlsanityData to 14 Oct 2026

Correlating Cross-Iteration Noise for DP-SGD using Model Curvature

arXiv 2510.05416 ↗First posted 6 Oct 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Differentially private stochastic gradient descent (DP-SGD) offers the promise of training deep learning models while mitigating many privacy risks. However, there is currently a large accuracy gap between DP-SGD and normal SGD training. This has resulted in different lines of research investigating orthogonal ways of improving privacy-preserving training. One such line of work, known as DP-MF, correlates the privacy noise across different iterations of stochastic gradient descent -- allowing later iterations to cancel out some of the noise added to earlier iterations. In this paper, we study how to improve this noise correlation. We propose a technique called NoiseCurve that uses model curvature, estimated from public unlabeled data, to improve the quality of this cross-iteration noise correlation. Our experiments on various datasets, models, and privacy parameters show that the noise correlations computed by NoiseCurve offer consistent and significant improvements in accuracy over the correlation scheme used by DP-MF.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

48 of 62 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2016Deep Learning with Differential Privacycited by 2,728
  12. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
36 more
  1. 2017Deep Image Priorcited by 1,098
  2. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  5. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  6. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  7. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  8. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  9. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 170
  10. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  11. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 90
  12. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 88
  13. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  14. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  15. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  16. 2022Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streamscited by 59
  17. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  18. 2022Multi-Epoch Matrix Factorization Mechanisms for Private Machine Learningcited by 48
  19. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 47
  20. 2020Privacy Amplification via Random Check-Inscited by 39
  21. 2018Concentrated Differentially Private Gradient Descent with Adaptive per-Iteration Privacy Budgetcited by 32
  22. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 24
  23. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  24. 2023Gradient Descent with Linearly Correlated Noise: Theory and Applications to Differential Privacycited by 18
  25. 2021Analysis of stochastic Lanczos quadrature for spectrum approximationcited by 17
  26. 2024Efficient and Near-Optimal Noise Generation for Streaming Differential Privacycited by 17
  27. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 16
  28. 2024Scalable DP-SGD: Shuffling vs. Poisson Subsamplingcited by 16
  29. 2025Privacy amplification by random allocationcited by 15
  30. 2025Correlated Noise Mechanisms for Differentially Private Learningcited by 14
  31. 2023Faster Differentially Private Convex Optimization via Second-Order Methodscited by 13
  32. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  33. 2025Back to Square Roots: An Optimal Bound on the Matrix Factorization Error for Multi-Epoch Differentially Private SGDcited by 7
  34. 2025On Design Principles for Private Adaptive Optimizerscited by 6
  35. 2025Tighter Privacy Analysis for Truncated Poisson Samplingcited by 5
  36. 2025An Inversion Theorem for Buffered Linear Toeplitz (BLT) Matrices and Applications to Streaming Differential Privacycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DP-λCGD: Efficient Noise Correlation for Differentially Private Model Trainingcited by 3