mlsanityData to 14 Oct 2026

How to Protect Copyright Data in Optimization of Large Language Models?

arXiv 2308.12247 ↗First posted 23 Aug 2023cs.LG cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) and generative AI have played a transformative role in computer research and applications. Controversy has arisen as to whether these models output copyrighted data, which can occur if the data the models are trained on is copyrighted. LLMs are built on the transformer neural network architecture, which in turn relies on a mathematical computation called Attention that uses the softmax function. In this paper, we show that large language model training and optimization can be seen as a softmax regression problem. We then establish a method of efficiently performing softmax regression, in a way that prevents the regression function from generating copyright data. This establishes a theoretical method of training large language models in a way that avoids generating copyright data.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 65 of its 96 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

65 of 96 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
38 more
  1. 2023A Watermark for Large Language Modelscited by 725
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  4. 2020A Survey on Visual Transformercited by 654
  5. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  7. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  8. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 502
  9. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  10. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  11. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  12. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  13. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  14. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
  15. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  16. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  17. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  18. 2023Representational Strengths and Limitations of Transformerscited by 133
  19. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  20. 2023MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformercited by 124
  21. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  22. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  23. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 113
  24. 2023Fast Attention Requires Bounded Entriescited by 102
  25. 2023On Provable Copyright Protection for Generative Modelscited by 98
  26. 2021Protecting Intellectual Property of Language Generation APIs with Lexical Watermarkcited by 84
  27. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  28. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  29. 2022CATER: Intellectual Property Protection on Text Generation APIs via Conditional Watermarkscited by 76
  30. 2023Federated Learning of Gboard Language Models with Differential Privacycited by 66
  31. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  32. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 59
  33. 2020Training (Overparametrized) Neural Networks in Near-Linear Timecited by 58
  34. 2023Attention Scheme Inspired Softmax Regressioncited by 49
  35. 2023An Over-parameterized Exponential Regressioncited by 45
  36. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  37. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  38. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 8; 2024: 14; 2025: 4.

26 papers.

Sort:

By area:

  1. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 43
  2. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 38
  3. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  4. 2024An Economic Solution to Copyright Challenges of Generative AIcited by 18
  5. 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?cited by 18
  6. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  7. 2024SoK: Dataset Copyright Auditing in Machine Learning Systemscited by 15
  8. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  9. 2023Fine-tune Language Models to Approximate Unbiased In-context Learningcited by 12
  10. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 12
  11. 2024Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokenscited by 11
  12. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
14 more
  1. 2024Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Surveycited by 8
  2. 2023The Expressibility of Polynomial based Attention Schemecited by 7
  3. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  4. 2023One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Spacecited by 6
  5. 2024Tackling Copyright Issues in AI Image Generation Through Originality Estimation and Genericizationcited by 6
  6. 2024Is Your AI Truly Yours? Leveraging Blockchain for Copyrights, Provenance, and Lineagecited by 4
  7. 2024Strong Copyright Protection for Language Models via Adaptive Model Fusioncited by 4
  8. 2025Label Smoothing Improves Gradient Ascent in LLM Unlearningcited by 3
  9. 2025Large Language Models as Search Engines: Societal Challengescited by 3
  10. 2023Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applicationscited by 2
  11. 2023Computational Copyright: Towards A Royalty Model for Music Generative AIcited by 2
  12. 2025Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiencycited by 2
  13. 2024Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AIcited by 1
  14. 2024Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergencecited by 0