mlsanityData to 7 Oct 2026

TDDBench: A Benchmark for Training data detection

arXiv 2411.03363 ↗First posted 5 Nov 2024cs.CR cs.LGcited by 0

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Training Data Detection (TDD) is a task aimed at determining whether a specific data instance is used to train a machine learning model. In the computer security literature, TDD is also referred to as Membership Inference Attack (MIA). Given its potential to assess the risks of training data breaches, ensure copyright authentication, and verify model unlearning, TDD has garnered significant attention in recent years, leading to the development of numerous methods. Despite these advancements, there is no comprehensive benchmark to thoroughly evaluate the effectiveness of TDD methods. In this work, we introduce TDDBench, which consists of 13 datasets spanning three data modalities: image, tabular, and text. We benchmark 21 different TDD methods across four detection paradigms and evaluate their performance from five perspectives: average detection performance, best detection performance, memory consumption, and computational efficiency in both time and memory. With TDDBench, researchers can identify bottlenecks and areas for improvement in TDD algorithms, while practitioners can make informed trade-offs between effectiveness and efficiency when selecting TDD algorithms for specific use cases. Our extensive experiments also reveal the generally unsatisfactory performance of TDD algorithms across different datasets. To enhance accessibility and reproducibility, we open-source TDDBench for the research community at https://github.com/zzh9568/TDDBench.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 61 of its 78 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

61 of 78 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks13 Mar
  2. 2024Do Membership Inference Attacks Work on Large Language Models?12 Feb
  3. 2023Low-Cost High-Power Membership Inference Attacks6 Dec
  4. 2023SoK: Comparing Different Membership Inference Attacks with a Comprehensive Benchmark12 Jul
  5. 2023Scalable Membership Inference Attacks via Quantile Regression7 Jul
  6. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  7. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries19 Oct
  8. 2022Membership-Doctor: Comprehensive Assessment of Membership Inference Against Machine Learning Models22 Aug
  9. 2022Debiasing Learning for Membership Inference Attacks Against Recommender Systems24 Jun
  10. 2021Membership Inference Attacks From First Principles7 Dec
  11. 2021Enhanced Membership Inference Attacks against Machine Learning Models18 Nov
  12. 2021On the Importance of Difficulty Calibration in Membership Inference Attacks15 Nov
3 more
  1. 2021Membership Inference Attacks Against Recommender Systems16 Sep
  2. 2021Dataset Inference: Ownership Resolution in Machine Learning21 Apr
  3. 2021Practical Blind Membership Inference Attack via Differential Comparisons5 Jan

Other references · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  11. 2016Wide Residual Networkscited by 3,057
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
34 more
  1. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  2. 2020Extracting Training Data from Large Language Modelscited by 2,144
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2017A Closer Look at Memorization in Deep Networkscited by 936
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  8. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  9. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  10. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  11. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  12. 2020Label-Only Membership Inference Attackscited by 371
  13. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  14. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  15. 2023Towards Unbounded Machine Unlearningcited by 261
  16. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  17. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  18. 2020When Machine Unlearning Jeopardizes Privacycited by 181
  19. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 177
  20. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  21. 2020Membership Leakage in Label-Only Exposurescited by 156
  22. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 136
  23. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  24. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 108
  25. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  26. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 83
  27. 2021Membership Inference Attack on Graph Neural Networkscited by 81
  28. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  29. 2020ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learningcited by 66
  30. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  31. 2021Adapting Membership Inference Attacks to GNN for Graph Classification: Approaches and Implicationscited by 50
  32. 2024Data Contamination Calibration for Black-box LLMscited by 23
  33. 2020Sampling Attacks: Amplification of Membership Inference Attacks by Repeated Queriescited by 15
  34. 2022Commonality in Natural Images Rescues GANs: Pretraining GANs with Generic and Privacy-free Synthetic Datacited by 6

Built on by

Not cited yet.