Footprints of Data in a Classifier: Understanding the Privacy Risks and Solution Strategies

The widespread deployment of Artificial Intelligence (AI) across government and private industries brings both advancements and heightened privacy and security concerns. Article 17 of the General Data Protection Regulation (GDPR) mandates the Right to Erasure, requiring data to be permanently removed from a system to prevent potential compromise. While existing research primarily focuses on erasing sensitive data attributes, several passive data compromise mechanisms remain underexplored and unaddressed. One such issue arises from the residual footprints of training data embedded within predictive models. Performance disparities between test and training data can inadvertently reveal which data points were part of the training set, posing a privacy risk. This study examines how two fundamental aspects of classifier systems - training data quality and classifier training methodology - contribute to privacy vulnerabilities. Our theoretical analysis demonstrates that classifiers exhibit universal vulnerability under conditions of data imbalance and distributional shifts. Empirical findings reinforce our theoretical results, highlighting the significant role of training data quality in classifier susceptibility. Additionally, our study reveals that a classifier's operational mechanism and architectural design impact its vulnerability. We further investigate mitigation strategies through data obfuscation techniques and analyze their impact on both privacy and classification performance. To aid practitioners, we introduce a privacy-performance trade-off index, providing a structured approach to balancing privacy protection with model effectiveness. The findings offer valuable insights for selecting classifiers and curating training data in diverse real-world applications.