# Building and curating conversational corpora for diversity-aware language science and technology

arXiv: https://arxiv.org/abs/2203.03399 · HTML: https://mlsanity.com/paper/2203.03399/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 144 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [ConvoKit: A Toolkit for the Analysis of Conversations](https://mlsanity.com/paper/2005.04246/) (2020)
- [Including Signed Languages in Natural Language Processing](https://mlsanity.com/paper/2105.05222/) (2021)
- [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021)
- [Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP](https://mlsanity.com/paper/2109.06598/) (2021)
- [Scattertext: a Browser-Based Tool for Visualizing how Corpora Differ](https://mlsanity.com/paper/1703.00565/) (2017)
- [Disfluencies and Human Speech Transcription Errors](https://mlsanity.com/paper/1904.04398/) (2019)
- [Towards decolonising computational sciences](https://mlsanity.com/paper/2009.14258/) (2020)
- [A Toolbox for Construction and Analysis of Speech Datasets](https://mlsanity.com/paper/2104.04896/) (2021)
- [What shall we do with an hour of data? Speech recognition for the un- and under-served languages of Common Voice](https://mlsanity.com/paper/2105.04674/) (2021)

## Built on by (most cited first, 2 of 2)

- [The timing bottleneck: Why timing and overlap are mission-critical for conversational user interfaces, speech recognition and dialogue systems](https://mlsanity.com/paper/2307.15493/) (2023)
- [Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models](https://mlsanity.com/paper/2509.20237/) (2025)

## Abstract (arXiv)

We present an analysis pipeline and best practice guidelines for building and curating corpora of everyday conversation in diverse languages. Surveying language documentation corpora and other resources that cover 67 languages and varieties from 28 phyla, we describe the compilation and curation process, specify minimal properties of a unified format for interactional data, and develop methods for quality control that take into account turn-taking and timing. Two case studies show the broad utility of conversational data for (i) charting human interactional infrastructure and (ii) tracing challenges and opportunities for current ASR solutions. Linguistically diverse conversational corpora can provide new insights for the language sciences and stronger empirical foundations for language technology.