mlsanityData to 7 Oct 2026

ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods

arXiv 2406.15968 ↗First posted 23 Jun 2024cs.CL cs.LGcited by 61

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

The rapid scaling of large language models (LLMs) has raised concerns about the transparency and fair use of the data used in their pretraining. Detecting such content is challenging due to the scale of the data and limited exposure of each instance during training. We propose ReCaLL (Relative Conditional Log-Likelihood), a novel membership inference attack (MIA) to detect LLMs' pretraining data by leveraging their conditional language modeling capabilities. ReCaLL examines the relative change in conditional log-likelihoods when prefixing target data points with non-member context. Our empirical findings show that conditioning member data on non-member prefixes induces a larger decrease in log-likelihood compared to non-member data. We conduct comprehensive experiments and show that ReCaLL achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach. Moreover, we conduct an in-depth analysis of LLMs' behavior with different membership contexts, providing insights into how LLMs leverage membership information for effective inference at both the sequence and token level.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 29 of its 39 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

29 of 39 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024LLM Dataset Inference: Did you train on my dataset?10 Jun
  2. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration10 Nov
  3. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  4. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks8 Mar
  5. 2021Membership Inference Attacks From First Principles7 Dec
  6. 2021On the Importance of Difficulty Calibration in Membership Inference Attacks15 Nov

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
11 more
  1. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  2. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  3. 2023Larger language models do in-context learning differentlycited by 386
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  5. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  6. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  7. 2023Privacy Auditing with One (1) Training Runcited by 119
  8. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  9. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  10. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
  11. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 29; 2026: 23.

61 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  3. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  4. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  5. 2024Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Modelscited by 21
  6. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20
  7. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 20
  8. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  9. 2025How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergencecited by 13
  10. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  11. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 9
  12. 2024Fine-tuning can Help Detect Pretraining Data from Large Language Modelscited by 7
38 more
  1. 2024Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Modelscited by 6
  2. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  3. 2025Auditing Data Membership in Reinforcement Learning With Verifiable Rewardscited by 6
  4. 2024A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Modelscited by 5
  5. 2025Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attackcited by 5
  6. 2026Window-based Membership Inference Attacks Against Fine-tuned Large Language Modelscited by 5
  7. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  8. 2025Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Modelscited by 4
  9. 2026AttenMIA: LLM Membership Inference Attack through Attention Signalscited by 4
  10. 2025Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysiscited by 3
  11. 2025In-Context Probing for Membership Inference in Fine-Tuned Language Modelscited by 3
  12. 2025Machine Text Detectors are Membership Inference Attackscited by 2
  13. 2025Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Modelscited by 2
  14. 2026Membership Inference on LLMs in the Wildcited by 2
  15. 2026CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Modelscited by 2
  16. 2025On Membership Inference Attacks in Knowledge Distillationcited by 1
  17. 2025LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Datacited by 1
  18. 2025(Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMscited by 1
  19. 2025Provable Training Data Identification for Large Language Modelscited by 1
  20. 2025Ensemble Privacy Defense for Knowledge-Intensive LLMs against Membership Inference Attackscited by 1
  21. 2026Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMscited by 1
  22. 2025RN-F: A Novel Approach for Mitigating Contaminated Data in Large Language Modelscited by 0
  23. 2025AnalogFed: Privacy-Preserving Discovery of Analog Circuits at Scale with Federated Generative AIcited by 0
  24. 2025Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Modelscited by 0
  25. 2025Auditing Information Disclosure During Large-Scale Gradient-Based Training via Gradient Uniquenesscited by 0
  26. 2025Fast-MIA: Efficient and Scalable Membership Inference for LLMscited by 0
  27. 2025Learning to Attack: Uncovering Privacy Risks in Sequential Data Releasescited by 0
  28. 2025EL-MIA: Quantifying Membership Inference Risks of Sensitive Entities in LLMscited by 0
  29. 2025REMIND: Input Loss Landscapes Reveal Residual Memorization in Post-Unlearning LLMscited by 0
  30. 2025As If We've Met Before: LLMs Exhibit Certainty in Recognizing Seen Filescited by 0
  31. 2025Lost in Modality: Evaluating the Effectiveness of Text-Based Membership Inference Attacks on Large Multimodal Modelscited by 0
  32. 2025On the Effectiveness of Membership Inference in Targeted Data Extraction from Large Language Modelscited by 0
  33. 2026PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detectioncited by 0
  34. 2026On the Evidentiary Limits of Membership Inference for Copyright Auditingcited by 0
  35. 2026Detecting RLVR Training Data via Structural Convergence of Reasoningcited by 0
  36. 2026ReproMIA: A Comprehensive Analysis of Model Reprogramming for Proactive Membership Inference Attackscited by 0
  37. 2026Combating Data Laundering in LLM Trainingcited by 0
  38. 2026Detecting Data Contamination in Large Language Modelscited by 0

50 most cited of 61 All 61 (JSON)