# Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources

arXiv: https://arxiv.org/abs/2201.10066 · HTML: https://mlsanity.com/paper/2201.10066/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 25 Jan 2022.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning](https://mlsanity.com/paper/1912.10389/) (2019)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021)
- [Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus](https://mlsanity.com/paper/2105.05241/) (2021)
- [Datasheet for the Pile](https://mlsanity.com/paper/2201.07311/) (2022)
- [Wildbook: Crowdsourcing, computer vision, and data science for conservation](https://mlsanity.com/paper/1710.08880/) (2017)
- [Masader: Metadata Sourcing for Arabic Text and Speech Data Resources](https://mlsanity.com/paper/2110.06744/) (2021)

## Built on by (most cited first, 11 of 11)

- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset](https://mlsanity.com/paper/2303.03915/) (2023)
- [Emergent and Predictable Memorization in Large Language Models](https://mlsanity.com/paper/2304.11158/) (2023)
- [The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI](https://mlsanity.com/paper/2310.16787/) (2023)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Consent in Crisis: The Rapid Decline of the AI Data Commons](https://mlsanity.com/paper/2407.14933/) (2024)
- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [Data Governance in the Age of Large-Scale Data-Driven Language Technology](https://mlsanity.com/paper/2206.03216/) (2022)
- [The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources](https://mlsanity.com/paper/2406.16746/) (2024)
- [NusaCrowd: A Call for Open and Reproducible NLP Research in Indonesian Languages](https://mlsanity.com/paper/2207.10524/) (2022)

## Abstract (arXiv)

In recent years, large-scale data collection efforts have prioritized the amount of data collected in order to improve the modeling capabilities of large language models. This prioritization, however, has resulted in concerns with respect to the rights of data subjects represented in data collections, particularly when considering the difficulty in interrogating these collections due to insufficient documentation and tools for analysis. Mindful of these pitfalls, we present our methodology for a documentation-first, human-centered data collection project as part of the BigScience initiative. We identified a geographically diverse set of target language groups (Arabic, Basque, Chinese, Catalan, English, French, Indic languages, Indonesian, Niger-Congo languages, Portuguese, Spanish, and Vietnamese, as well as programming languages) for which to collect metadata on potential data sources. To structure this effort, we developed our online catalogue as a supporting tool for gathering metadata through organized public hackathons. We present our development process; analyses of the resulting resource metadata, including distributions over languages, regions, and resource types; and our lessons learned in this endeavor.