# RelaxLoss: Defending Membership Inference Attacks without Losing Utility

arXiv: https://arxiv.org/abs/2207.05801 · HTML: https://mlsanity.com/paper/2207.05801/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [The Composition Theorem for Differential Privacy](https://mlsanity.com/paper/1311.0776/) (2013)
- [Systematic Evaluation of Privacy Risks of Machine Learning Models](https://mlsanity.com/paper/2003.10595/) (2020)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples](https://mlsanity.com/paper/1909.10594/) (2019)
- [Machine Learning with Membership Privacy using Adversarial Regularization](https://mlsanity.com/paper/1807.05852/) (2018)
- [Machine Learning Models that Remember Too Much](https://mlsanity.com/paper/1709.07886/) (2017)
- [PacGAN: The power of two samples in generative adversarial networks](https://mlsanity.com/paper/1712.04086/) (2017)
- [BackPACK: Packing more into backprop](https://mlsanity.com/paper/1912.10985/) (2019)
- [Practical Blind Membership Inference Attack via Differential Comparisons](https://mlsanity.com/paper/2101.01341/) (2021)
- [Efficient Per-Example Gradient Computations](https://mlsanity.com/paper/1510.01799/) (2015)
- [On the Effectiveness of Regularization Against Membership Inference Attacks](https://mlsanity.com/paper/2006.05336/) (2020)
- [On the Privacy Properties of GAN-generated Samples](https://mlsanity.com/paper/2206.01349/) (2022)
- [On the Difficulty of Membership Inference Attacks](https://mlsanity.com/paper/2005.13702/) (2020)

## Built on by (most cited first, 30 of 37)

- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Low-Cost High-Power Membership Inference Attacks](https://mlsanity.com/paper/2312.03262/) (2023)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning](https://mlsanity.com/paper/2305.11759/) (2023)
- [Evaluations of Machine Learning Privacy Defenses are Misleading](https://mlsanity.com/paper/2404.17399/) (2024)
- [On the Privacy Risk of In-context Learning](https://mlsanity.com/paper/2411.10512/) (2024)
- [Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment](https://mlsanity.com/paper/2407.06443/) (2024)
- [On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook](https://mlsanity.com/paper/2307.16680/) (2023)
- [Does Training with Synthetic Data Truly Protect Privacy?](https://mlsanity.com/paper/2502.12976/) (2025)
- [SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks](https://mlsanity.com/paper/2403.08481/) (2024)
- [Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble](https://mlsanity.com/paper/2506.13972/) (2025)
- [Privacy-Preserving Low-Rank Adaptation against Membership Inference Attacks for Latent Diffusion Models](https://mlsanity.com/paper/2402.11989/) (2024)
- [Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective](https://mlsanity.com/paper/2506.13009/) (2025)
- [Mitigating Privacy Risk in Membership Inference by Convex-Concave Loss](https://mlsanity.com/paper/2402.05453/) (2024)
- [Trustworthy AI: Safety, Bias, and Privacy -- A Survey](https://mlsanity.com/paper/2502.10450/) (2025)
- [Discriminative Adversarial Privacy: Balancing Accuracy and Membership Privacy in Neural Networks](https://mlsanity.com/paper/2306.03054/) (2023)
- [Diffence: Fencing Membership Privacy With Diffusion Models](https://mlsanity.com/paper/2312.04692/) (2023)
- [Center-Based Relaxed Learning Against Membership Inference Attacks](https://mlsanity.com/paper/2404.17674/) (2024)
- [Defending Membership Inference Attacks via Privacy-aware Sparsity Tuning](https://mlsanity.com/paper/2410.06814/) (2024)
- [Representation Magnitude has a Liability to Privacy Vulnerability](https://mlsanity.com/paper/2407.16164/) (2024)
- [Membership Privacy Risks of Sharpness Aware Minimization](https://mlsanity.com/paper/2310.00488/) (2023)
- [Do Fairness Interventions Come at the Cost of Privacy: Evaluations for Binary Classifiers](https://mlsanity.com/paper/2503.06150/) (2025)
- [How does Bayesian Sampling help Membership Inference Attacks?](https://mlsanity.com/paper/2503.07482/) (2025)
- [Better Membership Inference Privacy Measurement through Discrepancy](https://mlsanity.com/paper/2405.15140/) (2024)
- [CLMIA: Membership Inference Attacks via Unsupervised Contrastive Learning](https://mlsanity.com/paper/2411.11144/) (2024)
- [Dynamic Probabilistic Noise Injection for Membership Inference Defense](https://mlsanity.com/paper/2505.13362/) (2025)
- [Estimating Model-Level Membership Inference Vulnerability Without Reference Models](https://mlsanity.com/paper/2510.19773/) (2025)
- [Leverage Unlearning to Sanitize LLMs](https://mlsanity.com/paper/2510.21322/) (2025)
- [United We Defend: Collaborative Membership Inference Defenses in Federated Learning](https://mlsanity.com/paper/2601.06866/) (2026)

## Abstract (arXiv)

As a long-term threat to the privacy of training data, membership inference attacks (MIAs) emerge ubiquitously in machine learning models. Existing works evidence strong connection between the distinguishability of the training and testing loss distributions and the model's vulnerability to MIAs. Motivated by existing results, we propose a novel training framework based on a relaxed loss with a more achievable learning target, which leads to narrowed generalization gap and reduced privacy leakage. RelaxLoss is applicable to any classification model with added benefits of easy implementation and negligible overhead. Through extensive evaluations on five datasets with diverse modalities (images, medical data, transaction records), our approach consistently outperforms state-of-the-art defense mechanisms in terms of resilience against MIAs as well as model utility. Our defense is the first that can withstand a wide range of attacks while preserving (or even improving) the target model's utility. Source code is available at https://github.com/DingfanChen/RelaxLoss