The importance of feature preprocessing for differentially private linear optimization
Not on the map. Why?
Abstract (arXiv)
Training machine learning models with differential privacy (DP) has received increasing interest in recent years. One of the most popular algorithms for training differentially private models is differentially private stochastic gradient descent (DPSGD) and its variants, where at each step gradients are clipped and combined with some noise. Given the increasing usage of DPSGD, we ask the question: is DPSGD alone sufficient to find a good minimizer for every dataset under privacy constraints? Towards answering this question, we show that even for the simple case of linear classification, unlike non-private optimization, (private) feature preprocessing is vital for differentially private optimization. In detail, we first show theoretically that there exists an example where without feature preprocessing, DPSGD incurs an optimality gap proportional to the maximum Euclidean norm of features over all samples. We then propose an algorithm called DPSGD-F, which combines DPSGD with feature preprocessing and prove that for classification tasks, it incurs an optimality gap proportional to the diameter of the features maxx, x' ∈ D |x - x'|2. We finally demonstrate the practicality of our algorithm on image classification benchmarks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 19 Jul 2023.
Builds on
22 of 36 references.
References · 22
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2016Deep Learning with Differential Privacycited by 2,728
- 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
- 2021Differentially Private Fine-tuning of Language Modelscited by 400
- 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
- 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
- 2018Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoisingcited by 218
- 2019Private Stochastic Convex Optimization with Optimal Ratescited by 186
- 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 173
- 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 153
- 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
- 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 128
10 more
- 2017Finite Sample Differentially Private Confidence Intervalscited by 104
- 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 86
- 2022Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributionscited by 52
- 2021Non-Euclidean Differentially Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 51
- 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
- 2021Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settingscited by 45
- 2021Instance-optimal Mean Estimation Under Differential Privacycited by 37
- 2020Evading Curse of Dimensionality in Unconstrained Private GLMs via Private Gradient Descentcited by 27
- 2022Differentially Private Generalized Linear Models Revisitedcited by 17
- 2023Subset-Based Instance Optimality in Private Estimationcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.