# ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods

arXiv: https://arxiv.org/abs/2406.15968 · HTML: https://mlsanity.com/paper/2406.15968/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference](https://mlsanity.com/paper/1906.11798/) (2019)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [On the Importance of Difficulty Calibration in Membership Inference Attacks](https://mlsanity.com/paper/2111.08440/) (2021)
- [Privacy Auditing with One (1) Training Run](https://mlsanity.com/paper/2305.08846/) (2023)
- [LLM Dataset Inference: Did you train on my dataset?](https://mlsanity.com/paper/2406.06443/) (2024)
- [Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models](https://mlsanity.com/paper/2404.02936/) (2024)
- [Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration](https://mlsanity.com/paper/2311.06062/) (2023)
- [Backdoor Attacks for In-Context Learning with Language Models](https://mlsanity.com/paper/2307.14692/) (2023)
- [Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation](https://mlsanity.com/paper/2309.11765/) (2023)
- [DE-COP: Detecting Copyrighted Content in Language Models Training Data](https://mlsanity.com/paper/2402.09910/) (2024)

## Built on by (most cited first, 30 of 64)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)](https://mlsanity.com/paper/2406.17975/) (2024)
- [Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models](https://mlsanity.com/paper/2502.18943/) (2025)
- [Undesirable Memorization in Large Language Models: A Survey](https://mlsanity.com/paper/2410.02650/) (2024)
- [Scaling Up Membership Inference: When and How Attacks Succeed on Large Language Models](https://mlsanity.com/paper/2411.00154/) (2024)
- [Context-Aware Membership Inference Attacks against Pre-trained Large Language Models](https://mlsanity.com/paper/2409.13745/) (2024)
- [Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding](https://mlsanity.com/paper/2409.03363/) (2024)
- [How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence](https://mlsanity.com/paper/2502.00678/) (2025)
- [Detecting Training Data of Large Language Models via Expectation Maximization](https://mlsanity.com/paper/2410.07582/) (2024)
- [SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks](https://mlsanity.com/paper/2506.10424/) (2025)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [Fine-tuning can Help Detect Pretraining Data from Large Language Models](https://mlsanity.com/paper/2410.10880/) (2024)
- [Auditing Data Membership in Reinforcement Learning With Verifiable Rewards](https://mlsanity.com/paper/2511.14045/) (2025)
- [Self-Comparison for Dataset-Level Membership Inference in Large (Vision-)Language Models](https://mlsanity.com/paper/2410.13088/) (2024)
- [Window-based Membership Inference Attacks Against Fine-tuned Large Language Models](https://mlsanity.com/paper/2601.02751/) (2026)
- [A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models](https://mlsanity.com/paper/2412.13475/) (2024)
- [Beyond Data Privacy: New Privacy Risks for Large Language Models](https://mlsanity.com/paper/2509.14278/) (2025)
- [Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models](https://mlsanity.com/paper/2510.09259/) (2025)
- [Tag&Tab: Pretraining Data Detection in Large Language Models Using Keyword-Based Membership Inference Attack](https://mlsanity.com/paper/2501.08454/) (2025)
- [Membership Inference Attacks Against Fine-tuned Diffusion Language Models](https://mlsanity.com/paper/2601.20125/) (2026)
- [AttenMIA: LLM Membership Inference Attack through Attention Signals](https://mlsanity.com/paper/2601.18110/) (2026)
- [Neural Breadcrumbs: Membership Inference Attacks on LLMs Through Hidden State and Attention Pattern Analysis](https://mlsanity.com/paper/2509.05449/) (2025)
- [In-Context Probing for Membership Inference in Fine-Tuned Language Models](https://mlsanity.com/paper/2512.16292/) (2025)
- [Membership Inference on LLMs in the Wild](https://mlsanity.com/paper/2601.11314/) (2026)
- [CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models](https://mlsanity.com/paper/2606.17464/) (2026)
- [(Token-Level) InfoRMIA: Stronger Membership Inference and Memorization Assessment for LLMs](https://mlsanity.com/paper/2510.05582/) (2025)
- [Machine Text Detectors are Membership Inference Attacks](https://mlsanity.com/paper/2510.19492/) (2025)
- [Memories Retrieved from Many Paths: A Multi-Prefix Framework for Robust Detection of Training Data Leakage in Large Language Models](https://mlsanity.com/paper/2511.20799/) (2025)
- [When Reasoning Leaks Membership: Membership Inference Attack on Black-box Large Reasoning Models](https://mlsanity.com/paper/2601.13607/) (2026)
- [Detecting RLVR Training Data via Structural Convergence of Reasoning](https://mlsanity.com/paper/2602.11792/) (2026)

## Abstract (arXiv)

The rapid scaling of large language models (LLMs) has raised concerns about the transparency and fair use of the data used in their pretraining. Detecting such content is challenging due to the scale of the data and limited exposure of each instance during training. We propose ReCaLL (Relative Conditional Log-Likelihood), a novel membership inference attack (MIA) to detect LLMs' pretraining data by leveraging their conditional language modeling capabilities. ReCaLL examines the relative change in conditional log-likelihoods when prefixing target data points with non-member context. Our empirical findings show that conditioning member data on non-member prefixes induces a larger decrease in log-likelihood compared to non-member data. We conduct comprehensive experiments and show that ReCaLL achieves state-of-the-art performance on the WikiMIA dataset, even with random and synthetic prefixes, and can be further improved using an ensemble approach. Moreover, we conduct an in-depth analysis of LLMs' behavior with different membership contexts, providing insights into how LLMs leverage membership information for effective inference at both the sequence and token level.