mlsanityData to 7 Oct 2026

Consent in Crisis: The Rapid Decline of the AI Data Commons

arXiv 2407.14933 ↗First posted 20 Jul 2024cs.CL cs.AI cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

General-purpose artificial intelligence (AI) systems are built on massive swathes of public web data, assembled into corpora such as C4, RefinedWeb, and Dolma. To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora. Our audit of 14,000 web domains provides an expansive view of crawlable web data and how codified data use preferences are changing over time. We observe a proliferation of AI-specific clauses to limit use, acute differences in restrictions on AI developers, as well as general inconsistencies between websites' expressed intentions in their Terms of Service and their robots.txt. We diagnose these as symptoms of ineffective web protocols, not designed to cope with the widespread re-purposing of the internet for AI. Our longitudinal analyses show that in a single year (2023-2024) there has been a rapid crescendo of data restrictions from web sources, rendering ~5%+ of all tokens in C4, or 28%+ of the most actively maintained, critical sources in C4, fully restricted from use. For Terms of Service crawling restrictions, a full 45% of C4 is now restricted. If respected or enforced, these restrictions are rapidly biasing the diversity, freshness, and scaling laws for general-purpose AI systems. We hope to illustrate the emerging crises in data consent, for both developers and creators. The foreclosure of much of the open web will impact not only commercial AI, but also non-commercial AI and academic research.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 80 of its 154 references that are in the map. First posted on arXiv 20 Jul 2024.

Builds on

80 of 154 references.

References · 80

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
38 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2018Model Cards for Model Reportingcited by 1,068
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  13. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  14. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  15. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  16. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  17. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  18. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  19. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  20. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  21. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  22. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  23. 2023Scaling Data-Constrained Language Modelscited by 391
  24. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  25. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  26. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  27. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  28. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  29. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  30. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  31. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  32. 2021Challenges in Detoxifying Language Modelscited by 169
  33. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  34. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
  35. 2019Does Object Recognition Work for Everyone?cited by 149
  36. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  37. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  38. 2020Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructurecited by 138

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 23; 2026: 11.

45 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  3. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  4. 2025Authenticated Delegation and Authorized AI Agentscited by 63
  5. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  6. 2024Improving Model Evaluation using SMART Filtering of Benchmark Datasetscited by 20
  7. 2025A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignmentcited by 20
  8. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  9. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
  10. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  11. 2025How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformationcited by 17
  12. 2024Public Domain 12M: A Highly Aesthetic Image-Text Dataset with Novel Governance Mechanismscited by 15
33 more
  1. 2025Position: The Most Expensive Part of an LLM should be its Training Datacited by 15
  2. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  3. 2026On Safety Risks in Experience-Driven Self-Evolving Agentscited by 12
  4. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  5. 2025User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policiescited by 8
  6. 2024Toxicity of the Commons: Curating Open-Source Pre-Training Datacited by 7
  7. 2024DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learningcited by 6
  8. 2024SoK: Blockchain-Based Decentralized AI (DeAI)cited by 6
  9. 2025Craw4LLM: Efficient Web Crawling for LLM Pretrainingcited by 6
  10. 2024To Err is AI : A Case Study Informing LLM Flaw Reporting Practicescited by 5
  11. 2025The KL3M Data Project: Copyright-Clean Training Resources for Large Language Modelscited by 5
  12. 2025Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigationcited by 5
  13. 2025Societal Impacts Research Requires Benchmarks for Creative Composition Taskscited by 3
  14. 2025Beyond Public Access in LLM Pre-Training Datacited by 3
  15. 2026A Sustainable AI Economy Needs Data Deals That Work for Generatorscited by 3
  16. 2025CLLoRA: An Approach to Measure the Effects of the Context Length for LLM Fine-Tuningcited by 2
  17. 2025Do Data Valuations Make Good Data Prices?cited by 2
  18. 2025Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outscited by 2
  19. 2025DICE: Data Influence Cascade in Decentralized Learningcited by 2
  20. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  21. 2025How Do Data Owners Say No? A Case Study of Data Consent Mechanisms in Web-Scraped Vision-Language AI Training Datasetscited by 2
  22. 2026The ATOM Report: Measuring the Open Language Model Ecosystemcited by 2
  23. 2025Beyond Release: Access Considerations for Generative AI Systemscited by 1
  24. 2024AccessShare: Co-designing Data Access and Sharing with Blind Peoplecited by 0
  25. 2024Props for Machine-Learning Securitycited by 0
  26. 2026Yes, But Not Always. Generative AI Needs Nuanced Opt-incited by 0
  27. 2026Identifying AI Web Scrapers Using Canary Tokenscited by 0
  28. 2026The Agentic Web Requires New Normative Infrastructurecited by 0
  29. 2026Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillationcited by 0
  30. 2026Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corporacited by 0
  31. 2026terms.txt: A Consent and Compensation Protocol for Agentic Web Accesscited by 0
  32. 2026AX is the New AEOcited by 0
  33. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMscited by 0