mlsanityData to 7 Oct 2026

Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecture

arXiv 2410.04454 ↗First posted 6 Oct 2024cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) utilize extensive knowledge databases and show powerful text generation ability. However, their reliance on high-quality copyrighted datasets raises concerns about copyright infringements in generated texts. Current research often employs prompt engineering or semantic classifiers to identify copyrighted content, but these approaches have two significant limitations: (1) Challenging to identify which specific subdataset (e.g., works from particular authors) influences an LLM's output. (2) Treating the entire training database as copyrighted, hence overlooking the inclusion of non-copyrighted training data. We propose Inner-Probe, a lightweight framework designed to evaluate the influence of copyrighted sub-datasets on LLM-generated texts. Unlike traditional methods relying solely on text, we discover that the results of multi-head attention (MHA) during LLM output generation provide more effective information. Thus, Inner-Probe performs sub-dataset contribution analysis using a lightweight LSTM based network trained on MHA results in a supervised manner. Harnessing such a prior, Inner-Probe enables non-copyrighted text detection through a concatenated global projector trained with unsupervised contrastive learning. Inner-Probe demonstrates 3x improved efficiency compared to semantic model training in sub-dataset contribution analysis on Books3, achieves 15.04% - 58.7% higher accuracy over baselines on the Pile, and delivers a 0.104 increase in AUC for non-copyrighted data filtering.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 52 of its 73 references that are in the map. First posted on arXiv 6 Oct 2024.

Builds on

52 of 73 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
38 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  3. 2023A Watermark for Large Language Modelscited by 667
  4. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  5. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  6. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  7. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  8. 2023Large Language Model Unlearningcited by 288
  9. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  10. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  12. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 122
  13. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  14. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  15. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  16. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  17. 2021Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoderscited by 88
  18. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  19. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  20. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  21. 2023A Survey of Large Language Models Attributioncited by 71
  22. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  23. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 65
  24. 2023REMARK-LLM: A Robust and Efficient Watermarking Framework for Generative Large Language Modelscited by 64
  25. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoningcited by 62
  26. 2024LLMmap: Fingerprinting For Large Language Modelscited by 56
  27. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  28. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  29. 2023Separate the Wheat from the Chaff: Model Deficiency Unlearning via Parameter-Efficient Module Operationcited by 33
  30. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 33
  31. 2023Causal Interpretation of Self-Attention in Pre-Trained Transformerscited by 30
  32. 2023Can Copyright be Reduced to Privacy?cited by 24
  33. 2023Model Leeching: An Extraction Attack Targeting LLMscited by 22
  34. 2024An Economic Solution to Copyright Challenges of Generative AIcited by 18
  35. 2022Formalizing Human Ingenuity: A Quantitative Framework for Copyright Law's Substantial Similaritycited by 15
  36. 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?cited by 15
  37. 2023ChatGPT Hallucinates when Attributing Answerscited by 14
  38. 2022Enhancing Out-of-Distribution Detection in Natural Language Understanding via Implicit Layer Ensemblecited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Modelcited by 0