# Towards the Science of Security and Privacy in Machine Learning

arXiv: https://arxiv.org/abs/1611.03814 · HTML: https://mlsanity.com/paper/1611.03814/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2016.

## Built on by (most cited first, 30 of 89)

- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models](https://mlsanity.com/paper/2102.02551/) (2021)
- [Proof-of-Learning: Definitions and Practice](https://mlsanity.com/paper/2103.05633/) (2021)
- [Membership Inference Attack on Graph Neural Networks](https://mlsanity.com/paper/2101.06570/) (2021)
- [Bad Characters: Imperceptible NLP Attacks](https://mlsanity.com/paper/2106.09898/) (2021)
- [Probabilistic Margins for Instance Reweighting in Adversarial Training](https://mlsanity.com/paper/2106.07904/) (2021)
- [Membership Inference Attacks Against Recommender Systems](https://mlsanity.com/paper/2109.08045/) (2021)
- [Socially Responsible AI Algorithms: Issues, Purposes, and Challenges](https://mlsanity.com/paper/2101.02032/) (2021)
- [Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Ahead](https://mlsanity.com/paper/2101.02559/) (2021)
- [Quantifying and Mitigating Privacy Risks of Contrastive Learning](https://mlsanity.com/paper/2102.04140/) (2021)
- [Amplifying Membership Exposure via Data Poisoning](https://mlsanity.com/paper/2211.00463/) (2022)
- [Federated Learning for Healthcare Domain - Pipeline, Applications and Challenges](https://mlsanity.com/paper/2211.07893/) (2022)
- [Auditing Membership Leakages of Multi-Exit Networks](https://mlsanity.com/paper/2208.11180/) (2022)
- [Architectural Backdoors in Neural Networks](https://mlsanity.com/paper/2206.07840/) (2022)
- [Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey](https://mlsanity.com/paper/2410.23687/) (2024)
- [Backdoor Attacks Against Dataset Distillation](https://mlsanity.com/paper/2301.01197/) (2023)
- [No Privacy Left Outside: On the (In-)Security of TEE-Shielded DNN Partition for On-Device ML](https://mlsanity.com/paper/2310.07152/) (2023)
- [Understanding the Interaction of Adversarial Training with Noisy Labels](https://mlsanity.com/paper/2102.03482/) (2021)
- [Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework](https://mlsanity.com/paper/2109.09829/) (2021)
- [Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representations](https://mlsanity.com/paper/2202.13295/) (2022)
- [Membership-Doctor: Comprehensive Assessment of Membership Inference Against Machine Learning Models](https://mlsanity.com/paper/2208.10445/) (2022)
- [Robust learning under clean-label attack](https://mlsanity.com/paper/2103.00671/) (2021)
- [Adversarial Robustness for Tabular Data through Cost and Utility Awareness](https://mlsanity.com/paper/2208.13058/) (2022)
- [Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfitting](https://mlsanity.com/paper/2110.03135/) (2021)
- [Learning and Certification under Instance-targeted Poisoning](https://mlsanity.com/paper/2105.08709/) (2021)
- [ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2407.05365/) (2024)
- [A Survey of Graph Unlearning](https://mlsanity.com/paper/2310.02164/) (2023)
- [Improving Adversarial Robustness by Putting More Regularizations on Less Robust Samples](https://mlsanity.com/paper/2206.03353/) (2022)

## Abstract (arXiv)

Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics. ML is now pervasive---new systems and models are being deployed in every domain imaginable, leading to rapid and widespread deployment of software based inference and decision making. There is growing recognition that ML exposes new vulnerabilities in software systems, yet the technical community's understanding of the nature and extent of these vulnerabilities remains limited. We systematize recent findings on ML security and privacy, focusing on attacks identified on these systems and defenses crafted to date. We articulate a comprehensive threat model for ML, and categorize attacks and defenses within an adversarial framework. Key insights resulting from works both in the ML and security communities are identified and the effectiveness of approaches are related to structural elements of ML algorithms and the data used to train them. We conclude by formally exploring the opposing relationship between model accuracy and resilience to adversarial manipulation. Through these explorations, we show that there are (possibly unavoidable) tensions between model complexity, accuracy, and resilience that must be calibrated for the environments in which they will be used.