mlsanityData to 13 Oct 2026

Training Foundation Models as Data Compression: On Information, Model Weights and Copyright Law

arXiv 2407.13493 ↗First posted 18 Jul 2024cs.CY cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The training process of foundation models as for other classes of deep learning systems is based on minimizing the reconstruction error over a training set. For this reason, they are susceptible to the memorization and subsequent reproduction of training samples. In this paper, we introduce a training-as-compressing perspective, wherein the model's weights embody a compressed representation of the training data. From a copyright standpoint, this point of view implies that the weights can be considered a reproduction or, more likely, a derivative work of a potentially protected set of works. We investigate the technical and legal challenges that emerge from this framing of the copyright of outputs generated by foundation models, including their implications for practitioners and researchers. We demonstrate that adopting an information-centric approach to the problem presents a promising pathway for tackling these emerging complex legal issues.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 24 of its 58 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

24 of 58 references.

References · 24

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  5. 2020Extracting Training Data from Large Language Modelscited by 2,181
  6. 2015Deep Learning and the Information Bottleneck Principlecited by 796
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  8. 2023Extracting Training Data from Diffusion Modelscited by 729
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 710
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 497
  11. 2018Scalable agent alignment via reward modeling: a research directioncited by 443
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
12 more
  1. 2018Model compression via distillation and quantizationcited by 287
  2. 2023Language Modeling Is Compressioncited by 225
  3. 2022Talking About Large Language Modelscited by 175
  4. 2023Foundation Models and Fair Usecited by 116
  5. 2023On Provable Copyright Protection for Generative Modelscited by 97
  6. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 93
  7. 2022An Introduction to Neural Data Compressioncited by 79
  8. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  9. 2023On the Creativity of Large Language Modelscited by 57
  10. 2018DeepZip: Lossless Data Compression using Recurrent Neural Networkscited by 33
  11. 2021Copyright in Generative Deep Learningcited by 29
  12. 2024The Files are in the Computer: On Copyright, Memorization, and Generative AIcited by 20

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026DC-Merge: Improving Model Merging with Directional Consistencycited by 8
  2. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
  3. 2026KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptationcited by 0
  4. 2026R4Tun: LLM-guided adaptive segmental tunnel lining segmentation in point cloudscited by 0