mlsanityData to 7 Oct 2026

When Do Fewer Coordinates Suffice in DP-SGD?

arXiv 2606.04375 ↗First posted 3 Jun 2026cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Differentially private stochastic gradient descent (DP-SGD) injects noise into every updated coordinate, making the injected noise energy scale with the ambient parameter dimension (d). We ask when private training can update fewer coordinates without losing the signal needed for optimization. We propose textscTP-TopK (Two-Phase TopK DP-SGD), a two-phase method for coordinate-sparse private training without public data, in which a private warm-up phase identifies a coordinate support used to guide the main training phase. We give a criterion characterizing when coordinate restriction can be beneficial, show via a nonconvex stationarity bound that under this condition the relevant noise term scales with the active dimension (k) rather than the full parameter dimension (d), and provide a lower bound on the reliability of warm-up-based coordinate ranking. Experiments on MNIST, FMNIST, and CIFAR-10 show that learned coordinate supports can retain more gradient energy than size-matched random supports, with the largest gains when the active dimension is small and warm-up scores are informative.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

42 of 55 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2019Deep Leakage from Gradientscited by 1,004
  7. 2021Membership Inference Attacks From First Principlescited by 839
  8. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  9. 2017Renyi Differential Privacycited by 643
  10. 2023A Survey on Large Language Models for Recommendationcited by 378
  11. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  12. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
30 more
  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  3. 2020A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promisescited by 230
  4. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  5. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  6. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 183
  7. 2021Numerical Composition of Differential Privacycited by 174
  8. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  9. 2018An overview of deep learning in medical imaging focusing on MRIcited by 140
  10. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  11. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  12. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  13. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  14. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  15. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  16. 2019Hypothesis Testing Interpretations and Renyi Differential Privacycited by 84
  17. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  18. 2023(Amplified) Banded Matrix Factorization: A unified approach to private trainingcited by 49
  19. 2024Differentially Private Fine-Tuning of Diffusion Modelscited by 18
  20. 2024Near Exact Privacy Amplification for Matrix Mechanismscited by 17
  21. 2024DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reductioncited by 10
  22. 2021Improving Differentially Private SGD via Randomly Sparsified Gradientscited by 8
  23. 2024DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reductioncited by 8
  24. 2025Differential privacy for medical deep learning: methods, tradeoffs, and deployment implicationscited by 7
  25. 2023Sparsity-Preserving Differentially Private Training of Large Embedding Modelscited by 5
  26. 2023Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Modelscited by 4
  27. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  28. 2023Pre-Pruning and Gradient-Dropping Improve Differentially Private Image Classificationcited by 3
  29. 2025Do You Really Need Public Data? Surrogate Public Data for Differential Privacy on Tabular Datacited by 3
  30. 2025An Optimization Framework for Differentially Private Sparse Fine-Tuningcited by 2

Built on by

Not cited yet.