# Scalable Private Learning with PATE

arXiv: https://arxiv.org/abs/1802.08908 · HTML: https://mlsanity.com/paper/1802.08908/

## Key facts

- Cited by 269 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2018.

## Built on by (most cited first, 30 of 269)

- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](https://mlsanity.com/paper/2101.04535/) (2021)
- [Adaptive Machine Unlearning](https://mlsanity.com/paper/2106.04378/) (2021)
- [Reconstructing Training Data with Informed Adversaries](https://mlsanity.com/paper/2201.04845/) (2022)
- [Differentially Private Diffusion Models](https://mlsanity.com/paper/2210.09929/) (2022)
- [Measuring Forgetting of Memorized Training Examples](https://mlsanity.com/paper/2207.00099/) (2022)
- [Deep Learning with Label Differential Privacy](https://mlsanity.com/paper/2102.06062/) (2021)
- [Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation](https://mlsanity.com/paper/2309.11765/) (2023)
- [Large Scale Private Learning via Low-rank Reparametrization](https://mlsanity.com/paper/2106.09352/) (2021)
- [Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning](https://mlsanity.com/paper/2102.12677/) (2021)
- [Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challenges](https://mlsanity.com/paper/2201.08135/) (2022)
- [Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models](https://mlsanity.com/paper/2305.15594/) (2023)
- [FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning](https://mlsanity.com/paper/2102.02514/) (2021)
- [Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture](https://mlsanity.com/paper/2110.08324/) (2021)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [Selective Differential Privacy for Language Modeling](https://mlsanity.com/paper/2108.12944/) (2021)
- [Privacy-Preserving Machine Learning: Methods, Challenges and Directions](https://mlsanity.com/paper/2108.04417/) (2021)
- [Membership Inference Attacks Against Recommender Systems](https://mlsanity.com/paper/2109.08045/) (2021)
- [Mixed Differential Privacy in Computer Vision](https://mlsanity.com/paper/2203.11481/) (2022)
- [Membership Inference Attack Susceptibility of Clinical Language Models](https://mlsanity.com/paper/2104.08305/) (2021)
- [Antipodes of Label Differential Privacy: PATE and ALIBI](https://mlsanity.com/paper/2106.03408/) (2021)
- [Improved Certified Defenses against Data Poisoning with (Deterministic) Finite Aggregation](https://mlsanity.com/paper/2202.02628/) (2022)
- [Public Data-Assisted Mirror Descent for Private Model Training](https://mlsanity.com/paper/2112.00193/) (2021)
- [Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Data](https://mlsanity.com/paper/2109.11429/) (2021)
- [Differential Privacy and Fairness in Decisions and Learning Tasks: A Survey](https://mlsanity.com/paper/2202.08187/) (2022)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)

## Abstract (arXiv)

The rapid adoption of machine learning has increased concerns about the privacy implications of machine learning models trained on sensitive data, such as medical records or other personal information. To address those concerns, one promising approach is Private Aggregation of Teacher Ensembles, or PATE, which transfers to a "student" model the knowledge of an ensemble of "teacher" models, with intuitive privacy provided by training teachers on disjoint data and strong privacy guaranteed by noisy aggregation of teachers' answers. However, PATE has so far been evaluated only on simple classification tasks like MNIST, leaving unclear its utility when applied to larger-scale learning tasks and real-world datasets. In this work, we show how PATE can scale to learning tasks with large numbers of output classes and uncurated, imbalanced training data with errors. For this, we introduce new noisy aggregation mechanisms for teacher ensembles that are more selective and add less noise, and prove their tighter differential-privacy guarantees. Our new mechanisms build on two insights: the chance of teacher consensus is increased by using more concentrated noise and, lacking consensus, no answer need be given to a student. The consensus answers used are more likely to be correct, offer better intuitive privacy, and incur lower-differential privacy cost. Our evaluation shows our mechanisms improve on the original PATE on all measures, and scale to larger tasks with both high utility and very strong privacy (varepsilon < 1.0).