Can R'enyi Accountants Beat Lossless Accountants for DP-SGD?

Recent frameworks in differential privacy (DP), such as privacy loss distributions and f-DP, provide lossless representations of the privacy leakage of mechanisms. These approaches are often considered to yield tighter (varepsilon, δ)-DP guarantees than those obtained from relaxed notions such as R'enyi DP. But is that always the case? We study this question for differentially private stochastic gradient descent (DP-SGD) with fixed-size batches under replace adjacency. Our key observation is that in this setting, privacy accounting based on R'enyi DP can use a pair of worst-case distributions (i.e., a dominating pair) that differs from the one used in the lossless frameworks. In particular, we characterize the worst-case distributions as two-dimensional Gaussian mixtures and derive an improved R'enyi DP bound for DP-SGD. Numerical and asymptotic comparisons show that across various hyperparameter settings, a R'enyi DP accountant based on the new bound can yield better privacy guarantees than those obtained from the lossless frameworks.