# On the Impact of Uncertainty and Calibration on Likelihood-Ratio Membership Inference Attacks

arXiv: https://arxiv.org/abs/2402.10686 · HTML: https://mlsanity.com/paper/2402.10686/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 34 of its 58 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 30 of 34)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Systematic Evaluation of Privacy Risks of Machine Learning Models](https://mlsanity.com/paper/2003.10595/) (2020)
- [MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples](https://mlsanity.com/paper/1909.10594/) (2019)
- [Machine Learning with Membership Privacy using Adversarial Regularization](https://mlsanity.com/paper/1807.05852/) (2018)
- [Membership Leakage in Label-Only Exposures](https://mlsanity.com/paper/2007.15528/) (2020)
- [Tempered Sigmoid Activations for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2007.14191/) (2020)
- [Stealing Hyperparameters in Machine Learning](https://mlsanity.com/paper/1802.05351/) (2018)
- [Revisiting Membership Inference Under Realistic Assumptions](https://mlsanity.com/paper/2005.10881/) (2020)
- [Low-Cost High-Power Membership Inference Attacks](https://mlsanity.com/paper/2312.03262/) (2023)
- [Membership Inference Attacks by Exploiting Loss Trajectory](https://mlsanity.com/paper/2208.14933/) (2022)
- [Understanding Membership Inferences on Well-Generalized Learning Models](https://mlsanity.com/paper/1802.04889/) (2018)
- [A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement](https://mlsanity.com/paper/2204.09308/) (2022)
- [Scalable Membership Inference Attacks via Quantile Regression](https://mlsanity.com/paper/2307.03694/) (2023)
- [Privacy For Free: Wireless Federated Learning Via Uncoded Transmission With Adaptive Power Control](https://mlsanity.com/paper/2006.05459/) (2020)
- [Evaluating Machine Unlearning via Epistemic Uncertainty](https://mlsanity.com/paper/2208.10836/) (2022)
- [Disparate Vulnerability to Membership Inference Attacks](https://mlsanity.com/paper/1906.00389/) (2019)
- [Against Membership Inference Attack: Pruning is All You Need](https://mlsanity.com/paper/2008.13578/) (2020)
- [Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Prediction](https://mlsanity.com/paper/2307.01610/) (2023)
- [Set-valued classification -- overview via a unified framework](https://mlsanity.com/paper/2102.12318/) (2021)
- [On the Effectiveness of Regularization Against Membership Inference Attacks](https://mlsanity.com/paper/2006.05336/) (2020)
- [Randomized Quantization is All You Need for Differential Privacy in Federated Learning](https://mlsanity.com/paper/2306.11913/) (2023)
- [Purifier: Defending Data Inference Attacks via Transforming Confidence Scores](https://mlsanity.com/paper/2212.00612/) (2022)
- [Data and Model Dependencies of Membership Inference Attack](https://mlsanity.com/paper/2002.06856/) (2020)
- [Bounding Information Leakage in Machine Learning](https://mlsanity.com/paper/2105.03875/) (2021)

## Built on by (most cited first, 4 of 4)

- [Trustworthy AI Must Account for Interactions](https://mlsanity.com/paper/2504.07170/) (2025)
- [Better Membership Inference Privacy Measurement through Discrepancy](https://mlsanity.com/paper/2405.15140/) (2024)
- [AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective](https://mlsanity.com/paper/2603.24857/) (2026)
- [Membership Inference via Pairwise Likelihood Ratios](https://mlsanity.com/paper/2609.12367/) (2026)

## Abstract (arXiv)

In a membership inference attack (MIA), an attacker exploits the overconfidence exhibited by typical machine learning models to determine whether a specific data point was used to train a target model. In this paper, we analyze the performance of the likelihood ratio attack (LiRA) within an information-theoretical framework that allows the investigation of the impact of the aleatoric uncertainty in the true data generation process, of the epistemic uncertainty caused by a limited training data set, and of the calibration level of the target model. We compare three different settings, in which the attacker receives decreasingly informative feedback from the target model: confidence vector (CV) disclosure, in which the output probability vector is released; true label confidence (TLC) disclosure, in which only the probability assigned to the true label is made available by the model; and decision set (DS) disclosure, in which an adaptive prediction set is produced as in conformal prediction. We derive bounds on the advantage of an MIA adversary with the aim of offering insights into the impact of uncertainty and calibration on the effectiveness of MIAs. Simulation results demonstrate that the derived analytical bounds predict well the effectiveness of MIAs.