Data to 14 Oct 2026

Overlearning Reveals Sensitive Attributes

arXiv 1905.11742 ↗First posted 28 May 2019cs.LG cs.NE stat.MLcited by 76

Not on the map. Why?

Abstract (arXiv)

"Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize racestextemdash even races that are not represented in the training datatextemdash and identities. We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be "re-purposed" for a different, privacy-violating task even in the absence of the original training data. We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 16; 2022: 18; 2023: 14; 2024: 12; 2025: 10; 2026: 6.

76 papers.

Sort:

By area:

2021Graph Unlearningcited by 138
2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 128
2021Dataset Inference: Ownership Resolution in Machine Learningcited by 116
2022Vertical Federated Learning: Concepts, Advances and Challengescited by 108
2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Modelscited by 85
2021Node-Level Membership Inference Attacks Against Graph Neural Networkscited by 84
2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
2023Bounding Training Data Reconstruction in DP-SGDcited by 46
2021Survey: Leakage and Privacy at Inference Timecited by 44
2024Prompt Stealing Attacks Against Large Language Modelscited by 40
2021Fair Normalizing Flowscited by 38
2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 36

38 more

2023Integration of Large Language Models and Federated Learningcited by 31
2021Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputscited by 18
2022Membership-Doctor: Comprehensive Assessment of Membership Inference Against Machine Learning Modelscited by 16
2022Inferring Sensitive Attributes from Model Explanationscited by 15
2022Purifier: Defending Data Inference Attacks via Transforming Confidence Scorescited by 15
2024No Free Lunch Theorem for Privacy-Preserving LLM Inferencecited by 13
2022You Don't Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers' Private Personascited by 11
2023Marich: A Query-efficient Distributionally Equivalent Model Extraction Attack using Public Datacited by 11
2022Are Two Heads the Same as One? Identifying Disparate Treatment in Fair Neural Networkscited by 10
2022Semi-Leak: Membership Inference Attacks Against Semi-supervised Learningcited by 10
2021Fairness Properties of Face Recognition and Obfuscation Systemscited by 9
2022On the Alignment of Group Fairness with Attribute Privacycited by 9
2023AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Interpretable Modelscited by 9
2022How Does a Deep Learning Model Architecture Impact Its Privacy? A Comprehensive Study of Privacy Attacks on CNNs and Transformerscited by 8
2023Scalable Infomin Learningcited by 8
2023VFLAIR: A Research Library and Benchmark for Vertical Federated Learningcited by 8
2022Skellam Mixture Mechanism: a Novel Approach to Federated Learning with Differential Privacycited by 7
2024A Method to Facilitate Membership Inference Attacks in Deep Learning Modelscited by 7
202110 Security and Privacy Problems in Large Foundation Modelscited by 6
2023SoK: Unintended Interactions among Machine Learning Defenses and Riskscited by 6
2024A Survey of Privacy-Preserving Model Explanations: Privacy Risks, Attacks, and Countermeasurescited by 6
2021Unsupervised Information Obfuscation for Split Inference of Neural Networkscited by 5
2021MixNN: Protection of Federated Learning Against Inference Attacks by Mixing Neural Network Layerscited by 5
2022Dikaios: Privacy Auditing of Algorithmic Fairness via Attribute Inference Attackscited by 5
2024Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm?cited by 5
2025SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmarkcited by 5
2021Anti-Neuron Watermarking: Protecting Personal Data Against Unauthorized Neural Networkscited by 4
2024Analyzing Inference Privacy Risks Through Gradients in Machine Learningcited by 4
2021The Connection between Out-of-Distribution Generalization and Privacy of ML Modelscited by 3
2022PriMask: Cascadable and Collusion-Resilient Data Masking for Mobile Cloud Inferencecited by 3
2023SoK: Comparing Different Membership Inference Attacks with a Comprehensive Benchmarkcited by 3
2023Salted Inference: Enhancing Privacy while Maintaining Efficiency of Split Inference in Mobile Computingcited by 3
2024Privacy Evaluation Benchmarks for NLP Modelscited by 3
2024Subject Data Auditing via Source Inference Attack in Cross-Silo Federated Learningcited by 3
2025Cooperative Decentralized Backdoor Attacks on Vertical Federated Learningcited by 3
2022MLink: Linking Black-Box Models from Multiple Domains for Collaborative Inferencecited by 2
2022Holistic risk assessment of inference attacks in machine learningcited by 2
2023Anonymous Learning via Look-Alike Clustering: A Precise Analysis of Model Generalizationcited by 2

50 most cited of 76 All 76 (JSON)