# Practical Privacy Filters and Odometers with R'enyi Differential Privacy and Applications to Differentially Private Deep Learning

arXiv: https://arxiv.org/abs/2103.01379 · HTML: https://mlsanity.com/paper/2103.01379/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 6 of 6)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Individual Privacy Accounting via a Renyi Filter](https://mlsanity.com/paper/2008.11193/) (2020)
- [Privacy Odometers and Filters: Pay-as-you-Go Composition](https://mlsanity.com/paper/1605.08294/) (2016)
- [Accuracy First: Selecting a Differential Privacy Level for Accuracy-Constrained ERM](https://mlsanity.com/paper/1705.10829/) (2017)
- [Privacy Accounting and Quality Control in the Sage Differentially Private ML Platform](https://mlsanity.com/paper/1909.01502/) (2019)

## Built on by (most cited first, 12 of 12)

- [Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions](https://mlsanity.com/paper/2207.04380/) (2022)
- [Privacy-Preserving Retrieval-Augmented Generation with Differential Privacy](https://mlsanity.com/paper/2412.04697/) (2024)
- [Fully Adaptive Composition in Differential Privacy](https://mlsanity.com/paper/2203.05481/) (2022)
- [Individual Privacy Accounting for Differentially Private Stochastic Gradient Descent](https://mlsanity.com/paper/2206.02617/) (2022)
- [Individual Privacy Accounting with Gaussian Differential Privacy](https://mlsanity.com/paper/2209.15596/) (2022)
- [Adaptive Randomized Smoothing: Certified Adversarial Robustness for Multi-Step Defences](https://mlsanity.com/paper/2406.10427/) (2024)
- [Privacy Budget Scheduling](https://mlsanity.com/paper/2106.15335/) (2021)
- [Sensitivity analysis in differentially private machine learning using hybrid automatic differentiation](https://mlsanity.com/paper/2107.04265/) (2021)
- [On the Privacy of Selection Mechanisms with Gaussian Noise](https://mlsanity.com/paper/2402.06137/) (2024)
- [DPack: Efficiency-Oriented Privacy Budget Scheduling](https://mlsanity.com/paper/2212.13228/) (2022)
- [Optimizer Dynamics at the Edge of Stability with Differential Privacy](https://mlsanity.com/paper/2512.19019/) (2025)
- [Your Privacy Depends on Others: Collusion Vulnerabilities in Individual Differential Privacy](https://mlsanity.com/paper/2601.12922/) (2026)

## Abstract (arXiv)

Differential Privacy (DP) is the leading approach to privacy preserving deep learning. As such, there are multiple efforts to provide drop-in integration of DP into popular frameworks. These efforts, which add noise to each gradient computation to make it DP, rely on composition theorems to bound the total privacy loss incurred over this sequence of DP computations. However, existing composition theorems present a tension between efficiency and flexibility. Most theorems require all computations in the sequence to have a predefined DP parameter, called the privacy budget. This prevents the design of training algorithms that adapt the privacy budget on the fly, or that terminate early to reduce the total privacy loss. Alternatively, the few existing composition results for adaptive privacy budgets provide complex bounds on the privacy loss, with constants too large to be practical. In this paper, we study DP composition under adaptive privacy budgets through the lens of R'enyi Differential Privacy, proving a simpler composition theorem with smaller constants, making it practical enough to use in algorithm design. We demonstrate two applications of this theorem for DP deep learning: adapting the noise or batch size online to improve a model's accuracy within a fixed total privacy loss, and stopping early when fine-tuning a model to reduce total privacy loss.