mlsanityData to 7 Oct 2026

AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

arXiv 2401.06408 ↗First posted 12 Jan 2024cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Large language models' (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation. However, decisions around what data is retained or removed during this initial stage are under-scrutinized. In our work, we ground web text, which is a popular pretraining data source, to its social and geographic contexts. We create a new dataset of 10.3 million self-descriptions of website creators, and extract information about who they are and where they are from: their topical interests, social roles, and geographic affiliations. Then, we conduct the first study investigating how ten "quality" and English language identification (langID) filters affect webpages that vary along these social dimensions. Our experiments illuminate a range of implicit preferences in data curation: we show that some quality classifiers act like topical domain filters, and langID can overlook English content from some regions of the world. Overall, we hope that our work will encourage a new line of research on pretraining data curation practices and its social implications.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 45 of its 79 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

45 of 79 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2018Datasheets for Datasetscited by 1,371
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  10. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  11. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
33 more
  1. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  4. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  5. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  6. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  7. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  8. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  9. 2016FastText.zip: Compressing text classification modelscited by 331
  10. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  11. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  12. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  13. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  14. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  15. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  16. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  17. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 118
  18. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  19. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  20. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  21. 2020"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairnesscited by 56
  22. 2023"I'm fully who I am": Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generationcited by 56
  23. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 52
  24. 2022Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gendercited by 47
  25. 2017Community Identity and User Engagement in a Multi-Community Landscapecited by 18
  26. 2022Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publicationscited by 14
  27. 2018A Fast, Compact, Accurate Model for Language Identification of Codemixed Textcited by 11
  28. 2021An Empirical Exploration in Quality Filtering of Text Datacited by 9
  29. 2023MegaWika: Millions of reports and their sources across 50 diverse languagescited by 9
  30. 2022Richer Countries and Richer Representationscited by 7
  31. 2023Mordecai 3: A Neural Geoparser and Event Geocodercited by 6
  32. 2021A Method to Analyze Multiple Social Identities in Twitter Bioscited by 5
  33. 2023Measuring Dimensions of Self-Presentation in Twitter Bios and their Links to Misinformation Sharingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  2. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  3. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
  4. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 33
  5. 2024The Impossibility of Fair LLMscited by 22
  6. 2024Who's in and who's out? A case study of multimodal CLIP-filtering in DataCompcited by 15
  7. 2024On Classification with Large Language Models in Cultural Analyticscited by 9
  8. 2024KidLM: Advancing Language Models for Children -- Early Insights and Future Directionscited by 7
  9. 2024Data Feminism for AIcited by 6
  10. 2024Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMscited by 5
  11. 2025so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMscited by 4
  12. 2024Sound Check: Auditing Audio Datasetscited by 3
9 more
  1. 2025Enhancing LLMs via High-Knowledge Data Selectioncited by 2
  2. 2025What Are They Filtering Out? An Experimental Benchmark of Filtering Strategies for Harm Reduction in Pretraining Datasetscited by 1
  3. 2025Beyond Text: Characterizing Domain Expert Needs in Document Researchcited by 1
  4. 2025Analyzing values about gendered language reform in LLMs' revisionscited by 1
  5. 2026Characterizing Narrative Content in Web-scale LLM Pretraining Datacited by 1
  6. 2025The Case for Repeatable, Open, and Expert-Grounded Hallucination Benchmarks in Large Language Modelscited by 0
  7. 2026Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrailscited by 0
  8. 2026Unlocking Latent Value: Taxonomy-Guided Recovery of High-Performing Data from Low-Tier Web Corporacited by 0
  9. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 0