# DP-Auditorium: a Large Scale Library for Auditing Differential Privacy

arXiv: https://arxiv.org/abs/2307.05608 · HTML: https://mlsanity.com/paper/2307.05608/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2023.

## Built on by (most cited first, 11 of 11)

- [The Elusive Pursuit of Reproducing PATE-GAN: Benchmarking, Auditing, Debugging](https://mlsanity.com/paper/2406.13985/) (2024)
- [BadGD: A unified data-centric framework to identify gradient descent vulnerabilities](https://mlsanity.com/paper/2405.15979/) (2024)
- [The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditing](https://mlsanity.com/paper/2506.16666/) (2025)
- [Sequentially Auditing Differential Privacy](https://mlsanity.com/paper/2509.07055/) (2025)
- [Multi-Stakeholder Alignment in LLM-Powered Collaborative AI Systems: A Multi-Agent Framework for Intelligent Tutoring](https://mlsanity.com/paper/2510.23245/) (2025)
- [Regularized f-Divergence Kernel Tests](https://mlsanity.com/paper/2601.19755/) (2026)
- [Differentially Private Auditing Under Strategic Response](https://mlsanity.com/paper/2605.07674/) (2026)
- [PrivacyGuard: A Modular Framework for Privacy Auditing in Machine Learning](https://mlsanity.com/paper/2510.23427/) (2025)
- [One RNG to Rule Them All: How Randomness Becomes an Attack Vector in Machine Learning](https://mlsanity.com/paper/2602.09182/) (2026)
- [Optimal Guarantees for Auditing R'enyi Differentially Private Machine Learning](https://mlsanity.com/paper/2605.21938/) (2026)
- [Picture the Epsilon: Pursuing Identity-Level Privacy Guarantees for Images](https://mlsanity.com/paper/2608.17147/) (2026)

## Abstract (arXiv)

New regulations and increased awareness of data privacy have led to the deployment of new and more efficient differentially private mechanisms across public institutions and industries. Ensuring the correctness of these mechanisms is therefore crucial to ensure the proper protection of data. However, since differential privacy is a property of the mechanism itself, and not of an individual output, testing whether a mechanism is differentially private is not a trivial task. While ad hoc testing techniques exist under specific assumptions, no concerted effort has been made by the research community to develop a flexible and extendable tool for testing differentially private mechanisms. This paper introduces DP-Auditorium as a step advancing research in this direction. DP-Auditorium abstracts the problem of testing differential privacy into two steps: (1) measuring the distance between distributions, and (2) finding neighboring datasets where a mechanism generates output distributions maximizing such distance. From a technical point of view, we propose three new algorithms for evaluating the distance between distributions. While these algorithms are well-established in the statistics community, we provide new estimation guarantees that exploit the fact that we are only interested in verifying whether a mechanism is differentially private, and not in obtaining an exact estimate of the distance between two distributions. DP-Auditorium is easily extensible, as demonstrated in this paper by implementing a well-known approximate differential privacy testing algorithm into our library. We provide an extensive comparison to date of multiple testers across varying sample sizes and differential privacy parameters, demonstrating that there is no single tester that dominates all others, and that a combination of different techniques is required to ensure proper testing of mechanisms.