mlsanityData to 14 Oct 2026

Detecting Data Contamination in Large Language Models

arXiv 2604.19561 ↗First posted 21 Apr 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) utilize large amounts of data for their training, some of which may come from copyrighted sources. Membership Inference Attacks (MIA) aim to detect those documents and whether they have been included in the training corpora of the LLMs. The black-box MIAs require a significant amount of data manipulation; therefore, their comparison is often challenging. We study state-of-the-art (SOTA) MIAs under the black-box assumptions and compare them to each other using a unified set of datasets to determine if any of them can reliably detect membership under SOTA LLMs. In addition, a new method, called the Familiarity Ranking, was developed to showcase a possible approach to black-box MIAs, thereby giving LLMs more freedom in their expression to understand their reasoning better. The results indicate that none of the methods are capable of reliably detecting membership in LLMs, as shown by an AUC-ROC of approximately 0.5 for all methods across several LLMs. The higher TPR and FPR for more advanced LLMs indicate higher reasoning and generalizing capabilities, showcasing the difficulty of detecting membership in LLMs using black-box MIAs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 19 references that are in the map. First posted on arXiv 21 Apr 2026.

Builds on

11 of 19 references.

References · 11

  1. 2023Extracting Training Data from Diffusion Modelscited by 732
  2. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  3. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  4. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 74
  5. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 64
  6. 2024Localizing Paragraph Memorization in Language Modelscited by 31
  7. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  8. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  9. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20
  10. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  11. 2024Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokenscited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models
  2. 2024A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models
  3. 2026Membership Inference on LLMs in the Wild
  4. 2024MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector
  5. 2024Do Membership Inference Attacks Work on Large Language Models?
  6. 2024LLM Dataset Inference: Did you train on my dataset?
  7. 2025On the Effectiveness of Membership Inference in Targeted Data Extraction from Large Language Models
  8. 2024Nob-MIAs: Non-biased Membership Inference Attacks Assessment on Large Language Models with Ex-Post Dataset Construction