# Scalable Membership Inference Attacks via Quantile Regression

arXiv: https://arxiv.org/abs/2307.03694 · HTML: https://mlsanity.com/paper/2307.03694/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Retiring Adult: New Datasets for Fair Machine Learning](https://mlsanity.com/paper/2108.04884/) (2021)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Systematic Evaluation of Privacy Risks of Machine Learning Models](https://mlsanity.com/paper/2003.10595/) (2020)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Enhanced Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/2111.09679/) (2021)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Membership Leakage in Label-Only Exposures](https://mlsanity.com/paper/2007.15528/) (2020)
- [On the Importance of Difficulty Calibration in Membership Inference Attacks](https://mlsanity.com/paper/2111.08440/) (2021)
- [Revisiting Membership Inference Under Realistic Assumptions](https://mlsanity.com/paper/2005.10881/) (2020)
- [Understanding Membership Inferences on Well-Generalized Learning Models](https://mlsanity.com/paper/1802.04889/) (2018)
- [Batch Multivalid Conformal Prediction](https://mlsanity.com/paper/2209.15145/) (2022)
- [Practical Adversarial Multivalid Conformal Prediction](https://mlsanity.com/paper/2206.01067/) (2022)
- [Online Multivalid Learning: Means, Moments, and Prediction Intervals](https://mlsanity.com/paper/2101.01739/) (2021)
- [Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries](https://mlsanity.com/paper/2210.10750/) (2022)
- [The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitation](https://mlsanity.com/paper/2302.08507/) (2023)

## Built on by (most cited first, 30 of 65)

- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Low-Cost High-Power Membership Inference Attacks](https://mlsanity.com/paper/2312.03262/) (2023)
- [Auditing f-Differential Privacy in One Run](https://mlsanity.com/paper/2410.22235/) (2024)
- [MoPe: Model Perturbation-based Privacy Attacks on Language Models](https://mlsanity.com/paper/2310.14369/) (2023)
- [Reconstruction Attacks on Machine Unlearning: Simple Models are Vulnerable](https://mlsanity.com/paper/2405.20272/) (2024)
- [SeqMIA: Sequential-Metric Based Membership Inference Attack](https://mlsanity.com/paper/2407.15098/) (2024)
- [Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding](https://mlsanity.com/paper/2409.03363/) (2024)
- [Membership Inference Attacks on Diffusion Models via Quantile Regression](https://mlsanity.com/paper/2312.05140/) (2023)
- [Adversarial Machine Unlearning](https://mlsanity.com/paper/2406.07687/) (2024)
- [Privacy Attacks on Image AutoRegressive Models](https://mlsanity.com/paper/2502.02514/) (2025)
- [The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD](https://mlsanity.com/paper/2410.06186/) (2024)
- [Model Inversion Attacks: A Survey of Approaches and Countermeasures](https://mlsanity.com/paper/2411.10023/) (2024)
- [Do Parameters Reveal More than Loss for Membership Inference?](https://mlsanity.com/paper/2406.11544/) (2024)
- [GLiRA: Black-Box Membership Inference Attack via Knowledge Distillation](https://mlsanity.com/paper/2405.07562/) (2024)
- [A Method to Facilitate Membership Inference Attacks in Deep Learning Models](https://mlsanity.com/paper/2407.01919/) (2024)
- [Mitigating Privacy Risk in Membership Inference by Convex-Concave Loss](https://mlsanity.com/paper/2402.05453/) (2024)
- [Cascading and Proxy Membership Inference Attacks](https://mlsanity.com/paper/2507.21412/) (2025)
- [Chameleon: Increasing Label-Only Membership Leakage with Adaptive Poisoning](https://mlsanity.com/paper/2310.03838/) (2023)
- [A Reliable Cryptographic Framework for Empirical Machine Unlearning Evaluation](https://mlsanity.com/paper/2404.11577/) (2024)
- [Order of Magnitude Speedups for LLM Membership Inference](https://mlsanity.com/paper/2409.14513/) (2024)
- [Practical Bayes-Optimal Membership Inference Attacks](https://mlsanity.com/paper/2505.24089/) (2025)
- [On the Impact of Uncertainty and Calibration on Likelihood-Ratio Membership Inference Attacks](https://mlsanity.com/paper/2402.10686/) (2024)
- [Anonymity Unveiled: A Practical Framework for Auditing Data Use in Deep Learning Models](https://mlsanity.com/paper/2409.06280/) (2024)
- [Generating Synthetic Data with Formal Privacy Guarantees: State of the Art and the Road Ahead](https://mlsanity.com/paper/2503.20846/) (2025)
- [Enhancing One-run Privacy Auditing with Quantile Regression-Based Membership Inference](https://mlsanity.com/paper/2506.15349/) (2025)
- [Toward Efficient Inference Attacks: Shadow Model Sharing via Mixture-of-Experts](https://mlsanity.com/paper/2510.13451/) (2025)
- [A hierarchical approach for assessing the vulnerability of tree-based classification models to membership inference attack](https://mlsanity.com/paper/2502.09396/) (2025)
- [Has My System Prompt Been Used? Large Language Model Prompt Membership Inference](https://mlsanity.com/paper/2502.09974/) (2025)
- [Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis](https://mlsanity.com/paper/2509.05449/) (2025)
- [Imitative Membership Inference Attack](https://mlsanity.com/paper/2509.06796/) (2025)

## Abstract (arXiv)

Membership inference attacks are designed to determine, using black box access to trained models, whether a particular example was used in training or not. Membership inference can be formalized as a hypothesis testing problem. The most effective existing attacks estimate the distribution of some test statistic (usually the model's confidence on the true label) on points that were (and were not) used in training by training many shadow models -- i.e. models of the same architecture as the model being attacked, trained on a random subsample of data. While effective, these attacks are extremely computationally expensive, especially when the model under attack is large. We introduce a new class of attacks based on performing quantile regression on the distribution of confidence scores induced by the model under attack on points that are not used in training. We show that our method is competitive with state-of-the-art shadow model attacks, while requiring substantially less compute because our attack requires training only a single model. Moreover, unlike shadow model attacks, our proposed attack does not require any knowledge of the architecture of the model under attack and is therefore truly ``black-box". We show the efficacy of this approach in an extensive series of experiments on various datasets and model architectures.