# Auditing f-Differential Privacy in One Run

arXiv: https://arxiv.org/abs/2410.22235 · HTML: https://mlsanity.com/paper/2410.22235/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 50 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Builds on (most cited first, 30 of 40)

- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Enhanced Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/2111.09679/) (2021)
- [Auditing Differentially Private Machine Learning: How Private is Private SGD?](https://mlsanity.com/paper/2006.07709/) (2020)
- [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](https://mlsanity.com/paper/2101.04535/) (2021)
- [MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples](https://mlsanity.com/paper/1909.10594/) (2019)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Machine Learning with Membership Privacy using Adversarial Regularization](https://mlsanity.com/paper/1807.05852/) (2018)
- [Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference](https://mlsanity.com/paper/1906.11798/) (2019)
- [Reconstructing Training Data with Informed Adversaries](https://mlsanity.com/paper/2201.04845/) (2022)
- [On the Importance of Difficulty Calibration in Membership Inference Attacks](https://mlsanity.com/paper/2111.08440/) (2021)
- [Protection Against Reconstruction and Its Applications in Private Federated Learning](https://mlsanity.com/paper/1812.00984/) (2018)
- [Privacy Auditing with One (1) Training Run](https://mlsanity.com/paper/2305.08846/) (2023)
- [Optimal Accounting of Differential Privacy via Characteristic Function](https://mlsanity.com/paper/2106.08567/) (2021)
- [Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture](https://mlsanity.com/paper/2110.08324/) (2021)
- [Scalable Membership Inference Attacks via Quantile Regression](https://mlsanity.com/paper/2307.03694/) (2023)
- [TAN Without a Burn: Scaling Laws of DP-SGD](https://mlsanity.com/paper/2210.03403/) (2022)
- [Differentially Private Learning Needs Hidden State (Or Much Faster Convergence)](https://mlsanity.com/paper/2203.05363/) (2022)
- [Bounding Training Data Reconstruction in Private (Deep) Learning](https://mlsanity.com/paper/2201.12383/) (2022)
- [Bounding Training Data Reconstruction in DP-SGD](https://mlsanity.com/paper/2302.07225/) (2023)
- [Debugging Differential Privacy: A Case Study for Privacy Auditing](https://mlsanity.com/paper/2202.12219/) (2022)
- [Detecting Violations of Differential Privacy](https://mlsanity.com/paper/1805.10277/) (2018)
- [One-shot Empirical Privacy Estimation for Federated Learning](https://mlsanity.com/paper/2302.03098/) (2023)
- [Students Parrot Their Teachers: Membership Inference on Model Distillation](https://mlsanity.com/paper/2303.03446/) (2023)
- [Defending against Reconstruction Attacks with R'enyi Differential Privacy](https://mlsanity.com/paper/2202.07623/) (2022)

## Built on by (most cited first, 30 of 30)

- [How Well Can Differential Privacy Be Audited in One Run?](https://mlsanity.com/paper/2503.07199/) (2025)
- [To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling](https://mlsanity.com/paper/2411.10614/) (2024)
- [Optimizing Canaries for Privacy Auditing with Metagradient Descent](https://mlsanity.com/paper/2507.15836/) (2025)
- [The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditing](https://mlsanity.com/paper/2506.16666/) (2025)
- [Sequentially Auditing Differential Privacy](https://mlsanity.com/paper/2509.07055/) (2025)
- [Enhancing One-run Privacy Auditing with Quantile Regression-Based Membership Inference](https://mlsanity.com/paper/2506.15349/) (2025)
- [Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models](https://mlsanity.com/paper/2606.09401/) (2026)
- [Setting varepsilon is not the Issue in Differential Privacy](https://mlsanity.com/paper/2511.06305/) (2025)
- [JAX-Privacy: A library for differentially private machine learning](https://mlsanity.com/paper/2602.17861/) (2026)
- [A Unified Framework for Adversary-Aware Differential Privacy Bounds](https://mlsanity.com/paper/2507.08158/) (2025)
- [Causal Evaluation of Membership Inference Attacks](https://mlsanity.com/paper/2602.02819/) (2026)
- [Understanding Disclosure Risk in Differential Privacy with Applications to Noise Calibration and Auditing (Extended Version)](https://mlsanity.com/paper/2603.12142/) (2026)
- [Your Privacy Depends on Others: Collusion Vulnerabilities in Individual Differential Privacy](https://mlsanity.com/paper/2601.12922/) (2026)
- [Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs](https://mlsanity.com/paper/2604.18697/) (2026)
- [On Reliability of Membership Inference Vulnerability Evaluation](https://mlsanity.com/paper/2605.25819/) (2026)
- [Auditing of Unlearning Algorithms](https://mlsanity.com/paper/2607.05898/) (2026)
- [Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarios](https://mlsanity.com/paper/2412.01756/) (2024)
- [Auditing Differential Privacy in the Black-Box Setting](https://mlsanity.com/paper/2503.12045/) (2025)
- [PrivacyGuard: A Modular Framework for Privacy Auditing in Machine Learning](https://mlsanity.com/paper/2510.23427/) (2025)
- [Observational Auditing of Label Privacy](https://mlsanity.com/paper/2511.14084/) (2025)
- [Beyond Membership: Limitations of Add/Remove Adjacency in Differential Privacy](https://mlsanity.com/paper/2511.21804/) (2025)
- [Sequential Auditing for f-Differential Privacy](https://mlsanity.com/paper/2602.06518/) (2026)
- [Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learning](https://mlsanity.com/paper/2603.00811/) (2026)
- [DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy](https://mlsanity.com/paper/2604.15851/) (2026)
- [When Are Trade-Off Functions Testable from Finite Samples?](https://mlsanity.com/paper/2605.10774/) (2026)
- [Optimal Guarantees for Auditing R'enyi Differentially Private Machine Learning](https://mlsanity.com/paper/2605.21938/) (2026)
- [Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Run](https://mlsanity.com/paper/2605.27292/) (2026)
- [Advancing the State-of-the-Art in Empirical Privacy Auditing](https://mlsanity.com/paper/2606.10481/) (2026)
- [Let's Ask Gauss: Improved One-Run Privacy Auditing](https://mlsanity.com/paper/2606.12733/) (2026)
- [Natural Identifiers for Privacy and Data Audits in Large Language Models](https://mlsanity.com/paper/2606.24408/) (2026)

## Abstract (arXiv)

Empirical auditing has emerged as a means of catching some of the flaws in the implementation of privacy-preserving algorithms. Existing auditing mechanisms, however, are either computationally inefficient requiring multiple runs of the machine learning algorithms or suboptimal in calculating an empirical privacy. In this work, we present a tight and efficient auditing procedure and analysis that can effectively assess the privacy of mechanisms. Our approach is efficient; similar to the recent work of Steinke, Nasr, and Jagielski (2023), our auditing procedure leverages the randomness of examples in the input dataset and requires only a single run of the target mechanism. And it is more accurate; we provide a novel analysis that enables us to achieve tight empirical privacy estimates by using the hypothesized f-DP curve of the mechanism, which provides a more accurate measure of privacy than the traditional ε,δ differential privacy parameters. We use our auditing procure and analysis to obtain empirical privacy, demonstrating that our auditing procedure delivers tighter privacy estimates.