mlsanityData to 7 Oct 2026

Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs

arXiv 2604.18697 ↗First posted 20 Apr 2026cs.CR cs.CL cs.LGcited by 1

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Indistinguishability properties such as differential privacy bounds or low empirically measured membership inference are widely treated as proxies to show a model is sufficiently protected against broader memorization risks. However, we show that indistinguishability properties are neither sufficient nor necessary for preventing data extraction in LLM APIs. We formalize a privacy-game separation between extraction and indistinguishability-based privacy, showing that indistinguishability and inextractability are incomparable: upper-bounding distinguishability does not upper-bound extractability. To address this gap, we introduce (l, b)-inextractability as a definition that requires at least 2b expected queries for any black-box adversary to induce the LLM API to emit a protected l-gram substring. We instantiate this via a worst-case extraction game and derive a rank-based extraction risk upper bound for targeted exact extraction, as well as extensions to cover untargeted and approximate extraction. The resulting estimator captures the extraction risk over multiple attack trials and prefix adaptations. We show that it can provide a tight and efficient estimation for standard greedy extraction and an upper bound on the probabilistic extraction risk given any decoding configuration. We empirically evaluate extractability across different models, clarifying its connection to distinguishability, demonstrating its advantage over existing extraction risk estimators, and providing actionable mitigation guidelines across model training, API access, and decoding configurations in LLM API deployment. Our code is publicly available at: https://github.com/Emory-AIMS/Inextractability.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 37 of its 61 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

37 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training27 Feb
  2. 2024PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Traps14 Mar
  3. 2024Do Membership Inference Attacks Work on Large Language Models?12 Feb
  4. 2021Membership Inference Attacks From First Principles7 Dec

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  8. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  12. 2023Detecting Pretraining Data from Large Language Modelscited by 375
21 more
  1. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  2. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  3. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  4. 2022Reconstructing Training Data with Informed Adversariescited by 143
  5. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  6. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  7. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  8. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  9. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  10. 2022Are Attribute Inference Attacks Just Imputation?cited by 48
  11. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 47
  12. 2024Measuring memorization in language models via probabilistic extractioncited by 40
  13. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  14. 2024Auditing f-Differential Privacy in One Runcited by 28
  15. 2020Investigating Membership Inference Attacks under Data Dependenciescited by 27
  16. 2025Language Models May Verbatim Complete Text They Were Not Explicitly Trained Oncited by 20
  17. 2024Measuring memorization in RLHF for code completioncited by 12
  18. 2022Unsupervised Text Deidentificationcited by 11
  19. 2025Extracting alignment data in open modelscited by 7
  20. 2024Data Reconstruction: When You See It and When You Don'tcited by 6
  21. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Memorization Is Not Extraction: Tight Differential-Privacy Bounds and Audit Blind Spotscited by 0