mlsanityData to 7 Oct 2026

RECAP: Reproducing Copyrighted Data from LLMs Training with an Agentic Pipeline

arXiv 2510.25941 ↗First posted 29 Oct 2025cs.CLcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

If we cannot inspect the training data of a large language model (LLM), how can we ever know what it has seen? We believe the most compelling evidence arises when the model itself freely reproduces the target content. As such, we propose RECAP, an agentic pipeline designed to elicit and verify memorized training data from LLM outputs. At the heart of RECAP is a feedback-driven loop, where an initial extraction attempt is evaluated by a secondary language model, which compares the output against a reference passage and identifies discrepancies. These are then translated into minimal correction hints, which are fed back into the target model to guide subsequent generations. In addition, to address alignment-induced refusals, RECAP includes a jailbreaking module that detects and overcomes such barriers. We evaluate RECAP on EchoTrace, a new benchmark spanning over 30 full books, and the results show that RECAP leads to substantial gains over single-iteration approaches. For instance, with GPT-4.1, the average ROUGE-L score for the copyrighted text extraction improved from 0.38 to 0.47 - a nearly 24% increase.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 28 of its 41 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

28 of 41 references.

References · 28

  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  4. 2020Extracting Training Data from Large Language Modelscited by 2,144
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  7. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  8. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  9. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  10. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  11. 20242 OLMo 2 Furiouscited by 302
  12. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
16 more
  1. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  2. 2023Copyright Violations and Large Language Modelscited by 108
  3. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  4. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  5. 2023Improving Language Models via Plug-and-Play Retrieval Feedbackcited by 87
  6. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  7. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  8. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 65
  9. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 54
  10. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  11. 2024Copyright Traps for Large Language Modelscited by 36
  12. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 33
  13. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  14. 2025Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Modelscited by 8
  15. 2025STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasingscited by 8
  16. 2025DIS-CO: Discovering Copyrighted Content in VLMs Training Datacited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
  2. 2025SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearning
  3. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  4. 2026Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs
  5. 2026CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs
  6. 2023Large Language Model Unlearning
  7. 2026Latent Reuse in Agent Skills: Multi-modal Clone Detection at Ecosystem Scale
  8. 2026Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents