mlsanityData to 14 Oct 2026

From 2.1M Hours of Raw Broadcast Audio to Finnish Speech SSL: Confidence Filtering and Architectural Analysis at Scale

arXiv 2610.13102 ↗First posted 9 Oct 2026cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Broadcast archives offer scale and diversity for self-supervised learning (SSL) but contain substantial noisy, non-speech, and multilingual content. Prior work typically relies on voice activity detection or language identification (LID) to extract speech, ensuring presence rather than quality. Whether and how to filter unlabeled speech by quality for SSL, particularly from noisy broadcasts, remains underexplored. We present a large-scale Finnish SSL study starting from 2.1M hours of unconstrained broadcast audio. We propose automatic speech recognition (ASR) confidence as a proxy for pretraining data quality and show 3-7% relative WER improvements over LID-based filtering despite a smaller training set. We further address the lack of controlled architectural comparisons in speech SSL by evaluating Transformer, Macaron-Net, Conformer, and E-Branchformer at matched parameter budgets. Our models improve upon prior Finnish SSL baselines, particularly at Base scale and out-of-domain.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

38 of 45 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  9. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  12. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
26 more
  1. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  2. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  3. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  4. 2016FastText.zip: Compressing text classification modelscited by 335
  5. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 176
  6. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  7. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  8. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  9. 2019Self-Training for End-to-End Speech Recognitioncited by 129
  10. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 113
  11. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  12. 2024YODAS: Youtube-Oriented Dataset for Audio and Speechcited by 60
  13. 2024Towards Robust Speech Representation Learning for Thousands of Languagescited by 50
  14. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 47
  15. 2023The Impact of Depth on Compositional Generalization in Transformer Language Modelscited by 38
  16. 2025Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languagescited by 34
  17. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 18
  18. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 14
  19. 2022Lahjoita puhetta -- a large-scale corpus of spoken Finnish with some benchmarkscited by 7
  20. 2022Exploring Capabilities of Monolingual Audio Transformers using Large Datasets in Automatic Speech Recognition of Czechcited by 4
  21. 2023Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Modelscited by 4
  22. 2023Confidence-based Ensembles of End-to-End Speech Recognition Modelscited by 4
  23. 2022Hearing voices at the National Library -- a speech corpus and acoustic model for the Swedish languagecited by 3
  24. 2024A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for Automatic Speech Recognition in Multilingual Oral History Archivescited by 3
  25. 2023Leveraging In-the-Wild Data for Effective Self-Supervised Pretraining in Speaker Recognitioncited by 2
  26. 2025Self-supervised learning of speech representations with Dutch archival datacited by 1

Built on by

Not cited yet.