# Detecting Violations of Differential Privacy

arXiv: https://arxiv.org/abs/1805.10277 · HTML: https://mlsanity.com/paper/1805.10277/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2018.

## Built on by (most cited first, 30 of 39)

- [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](https://mlsanity.com/paper/2101.04535/) (2021)
- [Privacy Auditing with One (1) Training Run](https://mlsanity.com/paper/2305.08846/) (2023)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [One-shot Empirical Privacy Estimation for Federated Learning](https://mlsanity.com/paper/2302.03098/) (2023)
- [Bayesian Estimation of Differential Privacy](https://mlsanity.com/paper/2206.05199/) (2022)
- [Auditing f-Differential Privacy in One Run](https://mlsanity.com/paper/2410.22235/) (2024)
- [A General Framework for Auditing Differentially Private Machine Learning](https://mlsanity.com/paper/2210.08643/) (2022)
- [Unleashing the Power of Randomization in Auditing Differentially Private ML](https://mlsanity.com/paper/2305.18447/) (2023)
- [On a Utilitarian Approach to Privacy Preserving Text Generation](https://mlsanity.com/paper/2104.11838/) (2021)
- [Differentially Private Steering for Large Language Model Alignment](https://mlsanity.com/paper/2501.18532/) (2025)
- [The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD](https://mlsanity.com/paper/2410.06186/) (2024)
- [How reparametrization trick broke differentially-private text representation learning](https://mlsanity.com/paper/2202.12138/) (2022)
- [Quantum Pufferfish Privacy: A Flexible Privacy Framework for Quantum Systems](https://mlsanity.com/paper/2306.13054/) (2023)
- [How Well Can Differential Privacy Be Audited in One Run?](https://mlsanity.com/paper/2503.07199/) (2025)
- [To Shuffle or not to Shuffle: Auditing DP-SGD with Shuffling](https://mlsanity.com/paper/2411.10614/) (2024)
- [Optimizing Canaries for Privacy Auditing with Metagradient Descent](https://mlsanity.com/paper/2507.15836/) (2025)
- [Revisiting Differentially Private Hyper-parameter Tuning](https://mlsanity.com/paper/2402.13087/) (2024)
- [The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditing](https://mlsanity.com/paper/2506.16666/) (2025)
- [Sequentially Auditing Differential Privacy](https://mlsanity.com/paper/2509.07055/) (2025)
- [Enhancing One-run Privacy Auditing with Quantile Regression-Based Membership Inference](https://mlsanity.com/paper/2506.15349/) (2025)
- [ContextLeak: Auditing Leakage in Private In-Context Learning Methods](https://mlsanity.com/paper/2512.16059/) (2025)
- [Auditing Differential Privacy in High Dimensions with the Kernel Quantum R'enyi Divergence](https://mlsanity.com/paper/2205.13941/) (2022)
- [Probing the Transition to Dataset-Level Privacy in ML Models Using an Output-Specific and Data-Resolved Privacy Profile](https://mlsanity.com/paper/2306.15790/) (2023)
- [Detecting Violations of Differential Privacy for Quantum Algorithms](https://mlsanity.com/paper/2309.04819/) (2023)
- [QueryCheetah: Fast Automated Discovery of Attribute Inference Attacks Against Query-Based Systems](https://mlsanity.com/paper/2409.01992/) (2024)
- [Can Stochastic Gradient Langevin Dynamics Provide Differential Privacy for Deep Learning?](https://mlsanity.com/paper/2110.05057/) (2021)
- [Epsilon*: Privacy Metric for Machine Learning Models](https://mlsanity.com/paper/2307.11280/) (2023)
- [A Unified Framework for Adversary-Aware Differential Privacy Bounds](https://mlsanity.com/paper/2507.08158/) (2025)
- [Understanding Disclosure Risk in Differential Privacy with Applications to Noise Calibration and Auditing (Extended Version)](https://mlsanity.com/paper/2603.12142/) (2026)
- [Statistical Privacy Guarantees of Machine Learning Preprocessing Techniques](https://mlsanity.com/paper/2109.02496/) (2021)

## Abstract (arXiv)

The widespread acceptance of differential privacy has led to the publication of many sophisticated algorithms for protecting privacy. However, due to the subtle nature of this privacy definition, many such algorithms have bugs that make them violate their claimed privacy. In this paper, we consider the problem of producing counterexamples for such incorrect algorithms. The counterexamples are designed to be short and human-understandable so that the counterexample generator can be used in the development process -- a developer could quickly explore variations of an algorithm and investigate where they break down. Our approach is statistical in nature. It runs a candidate algorithm many times and uses statistical tests to try to detect violations of differential privacy. An evaluation on a variety of incorrect published algorithms validates the usefulness of our approach: it correctly rejects incorrect algorithms and provides counterexamples for them within a few seconds.