# Dirichlet Mechanism for Differentially Private KL Divergence Minimization

arXiv: https://arxiv.org/abs/2110.01984 · HTML: https://mlsanity.com/paper/2110.01984/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 3 Oct 2021.

## Builds on (most cited first, 10 of 10)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [A survey on datasets for fairness-aware machine learning](https://mlsanity.com/paper/2110.00530/) (2021)
- [The Discrete Gaussian for Differential Privacy](https://mlsanity.com/paper/2004.00010/) (2020)
- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [Privacy for Free: Posterior Sampling and Stochastic Gradient Monte Carlo](https://mlsanity.com/paper/1502.07645/) (2015)
- [On the Theory and Practice of Privacy-Preserving Bayesian Data Analysis](https://mlsanity.com/paper/1603.07294/) (2016)
- [Differentially Private Bayesian Inference for Exponential Families](https://mlsanity.com/paper/1809.02188/) (2018)
- [R'enyi Differential Privacy Mechanisms for Posterior Sampling](https://mlsanity.com/paper/1710.00892/) (2017)

## Built on by (most cited first, 2 of 2)

- [KL-regularization Itself is Differentially Private in Bandits and RLHF](https://mlsanity.com/paper/2505.18407/) (2025)
- [End-to-End Differential Privacy in Training Deep Neural Network Classifiers](https://mlsanity.com/paper/2607.19580/) (2026)

## Abstract (arXiv)

Given an empirical distribution f(x) of sensitive data x, we consider the task of minimizing F(y) = DKL (f(x)Vert y) over a probability simplex, while protecting the privacy of x. We observe that, if we take the exponential mechanism and use the KL divergence as the loss function, then the resulting algorithm is the Dirichlet mechanism that outputs a single draw from a Dirichlet distribution. Motivated by this, we propose a R'enyi differentially private (RDP) algorithm that employs the Dirichlet mechanism to solve the KL divergence minimization task. In addition, given f(x) as above and haty an output of the Dirichlet mechanism, we prove a probability tail bound on DKL (f(x)Vert haty), which is then used to derive a lower bound for the sample complexity of our RDP algorithm. Experiments on real-world datasets demonstrate advantages of our algorithm over Gaussian and Laplace mechanisms in supervised classification and maximum likelihood estimation.