mlsanityData to 7 Oct 2026

Framework for Curating Speech Datasets and Evaluating ASR Systems: A Case Study for Polish

arXiv 2408.00005 ↗First posted 18 Jul 2024eess.AS cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Speech datasets available in the public domain are often underutilized because of challenges in discoverability and interoperability. A comprehensive framework has been designed to survey, catalog, and curate available speech datasets, which allows replicable evaluation of automatic speech recognition (ASR) systems. A case study focused on the Polish language was conducted; the framework was applied to curate more than 24 datasets and evaluate 25 combinations of ASR systems and models. This research constitutes the most extensive comparison to date of both commercial and free ASR systems for the Polish language. It draws insights from 600 system-model-test set evaluations, marking a significant advancement in both scale and comprehensiveness. The results of surveys and performance comparisons are available as interactive dashboards (https://huggingface.co/spaces/amu-cai/pl-asr-leaderboard) along with curated datasets (https://huggingface.co/datasets/amu-cai/pl-asr-bigos-v2, https://huggingface.co/datasets/pelcra/pl-asr-pelcra-for-bigos) and the open challenge call (https://poleval.pl/tasks/task3). Tools used for evaluation are open-sourced (https://github.com/goodmike31/pl-asr-bigos-tools), facilitating replication and adaptation for other languages, as well as continuous expansion with new datasets and systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 34 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

16 of 34 references.

References · 16

  1. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  2. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 467
  3. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  4. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  5. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  6. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  7. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  8. 2017PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparisoncited by 139
  9. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 93
  10. 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
  11. 2020WER we are and WER we think we arecited by 19
  12. 2022Accented Speech Recognition: Benchmarking, Pre-training, and Diverse Datacited by 15
4 more
  1. 2024Anatomy of Industrial Scale Multilingual ASRcited by 14
  2. 2021Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usabilitycited by 12
  3. 2024Toward Practical Automatic Speech Recognition and Post-Processing: a Call for Explainable Error Benchmark Guidelinecited by 4
  4. 2023SpokesBiz -- an Open Corpus of Conversational Polishcited by 2

Built on by

Not cited yet.