# Investigating data partitioning strategies for crosslinguistic low-resource ASR evaluation

arXiv: https://arxiv.org/abs/2208.12888 · HTML: https://mlsanity.com/paper/2208.12888/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 26 Aug 2022.

## Builds on (most cited first, 8 of 8)

- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets](https://mlsanity.com/paper/2008.02637/) (2020)
- [We Need to Talk About Random Splits](https://mlsanity.com/paper/2005.00636/) (2020)
- [RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions](https://mlsanity.com/paper/2005.03271/) (2020)
- [Leveraging End-to-End ASR for Endangered Language Documentation: An Empirical Study on Yolox'ochitl Mixtec](https://mlsanity.com/paper/2101.10877/) (2021)
- [BembaSpeech: A Speech Recognition Corpus for the Bemba Language](https://mlsanity.com/paper/2102.04889/) (2021)
- [Replicating and Extending "Because Their Treebanks Leak": Graph Isomorphism, Covariants, and Parser Performance](https://mlsanity.com/paper/2106.00352/) (2021)

## Built on by (most cited first, 7 of 7)

- [The Zeno's Paradox of `Low-Resource' Languages](https://mlsanity.com/paper/2410.20817/) (2024)
- [Breaking the Transcription Bottleneck: Fine-tuning ASR Models for Extremely Low-Resource Fieldwork Languages](https://mlsanity.com/paper/2506.17459/) (2025)
- [Frustratingly Easy Data Augmentation for Low-Resource ASR](https://mlsanity.com/paper/2509.15373/) (2025)
- [The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation](https://mlsanity.com/paper/2404.09371/) (2024)
- [Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual models](https://mlsanity.com/paper/2402.05581/) (2024)
- [Style or Signature? Artist-Disjoint Evaluation of Style Classification in Frozen Vision Embeddings](https://mlsanity.com/paper/2608.14435/) (2026)
- [Error Rate is Not Enough: Costs of Augmentation in Low-Resource ASR](https://mlsanity.com/paper/2610.13807/) (2026)

## Abstract (arXiv)

Many automatic speech recognition (ASR) data sets include a single pre-defined test set consisting of one or more speakers whose speech never appears in the training set. This "hold-speaker(s)-out" data partitioning strategy, however, may not be ideal for data sets in which the number of speakers is very small. This study investigates ten different data split methods for five languages with minimal ASR training resources. We find that (1) model performance varies greatly depending on which speaker is selected for testing; (2) the average word error rate (WER) across all held-out speakers is comparable not only to the average WER over multiple random splits but also to any given individual random split; (3) WER is also generally comparable when the data is split heuristically or adversarially; (4) utterance duration and intensity are comparatively more predictive factors of variability regardless of the data split. These results suggest that the widely used hold-speakers-out approach to ASR data partitioning can yield results that do not reflect model performance on unseen data or speakers. Random splits can yield more reliable and generalizable estimates when facing data sparsity.