mlsanityData to 14 Oct 2026

Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset

arXiv 2207.00220 ↗First posted 1 Jul 2022cs.CL cs.CYcited by 113

Not on the map. Why?

Abstract (arXiv)

One concern with the rise of large language models lies with their potential for significant harm, particularly from pretraining on biased, obscene, copyrighted, and private information. Emerging ethical approaches have attempted to filter pretraining material, but such approaches have been ad hoc and failed to take context into account. We offer an approach to filtering grounded in law, which has directly addressed the tradeoffs in filtering material. First, we gather and make available the Pile of Law, a 256GB (and growing) dataset of open-source English-language legal and administrative data, covering court opinions, contracts, administrative rules, and legislative records. Pretraining on the Pile of Law may help with legal tasks that have the promise to improve access to justice. Second, we distill the legal norms that governments have developed to constrain the inclusion of toxic or private content into actionable lessons for researchers and discuss how our dataset reflects these norms. Third, we show how the Pile of Law offers researchers the opportunity to learn such filtering rules directly from the data, providing an exciting new research direction in model-based processing.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Builds on 51 of its 135 references that are in the map. First posted on arXiv 1 Jul 2022.

Builds on

51 of 135 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2018Model Cards for Model Reportingcited by 1,080
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
38 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  2. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  4. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  6. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  7. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 349
  8. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  9. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  10. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  11. 2020The Pushshift Reddit Datasetcited by 292
  12. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  13. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  14. 2020Information Leakage in Embedding Modelscited by 203
  15. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  16. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  17. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  18. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  19. 2019Large-Scale Multi-Label Text Classification on EU Legislationcited by 104
  20. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  21. 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
  22. 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 77
  23. 2018Towards Robust and Privacy-preserving Text Representationscited by 75
  24. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 73
  25. 2022Do Language Models Plagiarize?cited by 73
  26. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 67
  27. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  28. 2018All You Need is "Love": Evading Hate-speech Detectioncited by 56
  29. 2018Challenges for Toxic Comment Classification: An In-Depth Error Analysiscited by 56
  30. 2018Privacy-preserving Neural Representations of Textcited by 54
  31. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 53
  32. 2022Data Governance in the Age of Large-Scale Data-Driven Language Technologycited by 33
  33. 2021ADePT: Auto-encoder based Differentially Private Text Transformationcited by 31
  34. 2018LexNLP: Natural language processing and information extraction for legal and regulatory textscited by 28
  35. 2020A Benchmark for Lease Contract Reviewcited by 26
  36. 2021Context-Aware Legal Citation Recommendation using Deep Learningcited by 12
  37. 2017Understanding State Preferences With Text As Data: Introducing the UN General Debate Corpuscited by 11
  38. 2020Research Challenges in Designing Differentially Private Text Generation Mechanismscited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 26; 2024: 36; 2025: 24; 2026: 22.

113 papers.

Sort:

By area:

  1. 2023Scaling Data-Constrained Language Modelscited by 403
  2. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  3. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  4. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  5. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  6. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  7. 2023Foundation Models and Fair Usecited by 116
  8. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  9. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  10. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  11. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  12. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
38 more
  1. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  2. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 60
  3. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  4. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
  5. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 50
  6. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 45
  7. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  8. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 41
  9. 2022Pre-trained Language Models for the Legal Domain: A Case Study on Indian Lawcited by 39
  10. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  11. 2024Nearest Neighbor Speculative Decoding for LLM Generation and Attributioncited by 25
  12. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  13. 2024NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helpscited by 20
  14. 2024Lossless data compression by large modelscited by 20
  15. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 20
  16. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17
  17. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  18. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Supportcited by 16
  19. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Expertscited by 16
  20. 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 16
  21. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  22. 2024Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?cited by 15
  23. 2023Large Language Models are legal but they are not: Making the case for a powerful LegalLLMcited by 12
  24. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  25. 2022Incorporating Context into Subword Vocabulariescited by 11
  26. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  27. 2024Data, Data Everywhere: A Guide for Pretraining Dataset Constructioncited by 11
  28. 2025L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Auditcited by 11
  29. 2023Unifying Corroborative and Contributive Attributions in Large Language Modelscited by 10
  30. 2024LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domaincited by 10
  31. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 10
  32. 2025Large Language Models Meet Legal Artificial Intelligence: A Surveycited by 10
  33. 2025Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Servingcited by 10
  34. 2023Legal Extractive Summarization of U.S. Court Opinionscited by 9
  35. 2023LexGPT 0.1: pre-trained GPT-J models with Pile of Lawcited by 9
  36. 2024ProgressGym: Alignment with a Millennium of Moral Progresscited by 9
  37. 2025DivScore: Zero-Shot Detection of LLM-Generated Text in Specialized Domainscited by 9
  38. 2023Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standardscited by 8

50 most cited of 113 All 113 (JSON)