# Building Better: Avoiding Pitfalls in Developing Language Resources when Data is Scarce

arXiv: https://arxiv.org/abs/2410.12691 · HTML: https://mlsanity.com/paper/2410.12691/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 55 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 19 of 19)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Power to the People? Opportunities and Challenges for Participatory AI](https://mlsanity.com/paper/2209.07572/) (2022)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP](https://mlsanity.com/paper/2109.06598/) (2021)
- [Towards Afrocentric NLP for African Languages: Where We Are and Where We Can Go](https://mlsanity.com/paper/2203.08351/) (2022)
- [What a Creole Wants, What a Creole Needs](https://mlsanity.com/paper/2206.00437/) (2022)
- [Best Practices in the Creation and Use of Emotion Lexicons](https://mlsanity.com/paper/2210.07206/) (2022)
- [The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Research](https://mlsanity.com/paper/2305.02797/) (2023)
- [Ethical Considerations for Machine Translation of Indigenous Languages: Giving a Voice to the Speakers](https://mlsanity.com/paper/2305.19474/) (2023)
- [Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources](https://mlsanity.com/paper/2211.15649/) (2022)
- [A Material Lens on Coloniality in NLP](https://mlsanity.com/paper/2311.08391/) (2023)
- [What Do Dialect Speakers Want? A Survey of Attitudes Towards Language Technology for German Dialects](https://mlsanity.com/paper/2402.11968/) (2024)
- [SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languages](https://mlsanity.com/paper/2402.08638/) (2024)
- [The Call for Socially Aware Language Technologies](https://mlsanity.com/paper/2405.02411/) (2024)
- ["It's how you do things that matters": Attending to Process to Better Serve Indigenous Communities with Language Technologies](https://mlsanity.com/paper/2402.02639/) (2024)
- [Why AI Is WEIRD and Should Not Be This Way: Towards AI For Everyone, With Everyone, By Everyone](https://mlsanity.com/paper/2410.16315/) (2024)
- [Human-Centric Research for NLP: Towards a Definition and Guiding Questions](https://mlsanity.com/paper/2207.04447/) (2022)

## Built on by (most cited first, 4 of 4)

- [Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia](https://mlsanity.com/paper/2503.07920/) (2025)
- [SAFARI: A Community-Engaged Approach and Dataset of Stereotype Resources in the Sub-Saharan African Context](https://mlsanity.com/paper/2602.22404/) (2026)
- [SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures](https://mlsanity.com/paper/2605.02601/) (2026)
- [Beyond Good Intentions: When Does the Framing of Multilingual and Low-Resource NLP Research Become a Caricature?](https://mlsanity.com/paper/2608.30866/) (2026)

## Abstract (arXiv)

Language is a form of symbolic capital that affects people's lives in many ways (Bourdieu1977,1991). As a powerful means of communication, it reflects identities, cultures, traditions, and societies more broadly. Therefore, data in a given language should be regarded as more than just a collection of tokens. Rigorous data collection and labeling practices are essential for developing more human-centered and socially aware technologies. Although there has been growing interest in under-resourced languages within the NLP community, work in this area faces unique challenges, such as data scarcity and limited access to qualified annotators. In this paper, we collect feedback from individuals directly involved in and impacted by NLP artefacts for medium- and low-resource languages. We conduct both quantitative and qualitative analyses of their responses and highlight key issues related to: (1) data quality, including linguistic and cultural appropriateness; and (2) the ethics of common annotation practices, such as the misuse of participatory research. Based on these findings, we make several recommendations for creating high-quality language artefacts that reflect the cultural milieu of their speakers, while also respecting the dignity and labor of data workers.