XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words
Not on the map. Why?
Abstract (arXiv)
Due to the absence of explicit word boundaries in the speech stream, the task of segmenting spoken sentences into word units without text supervision is particularly challenging. In this work, we leverage the most recent self-supervised speech models that have proved to quickly adapt to new tasks through fine-tuning, even in low resource conditions. Taking inspiration from semi-supervised learning, we fine-tune an XLS-R model to predict word boundaries themselves produced by top-tier speech segmentation systems: DPDP, VG-HuBERT, GradSeg and DP-Parse. Once XLS-R is fine-tuned, it is used to infer new word boundary labels that are used in turn for another fine-tuning step. Our method consistently improves the performance of each system and sets a new state-of-the-art that is, on average 130% higher than the previous one as measured by the F1 score on correctly discovered word tokens on five corpora featuring different languages. Finally, our system can segment speech from languages unseen during fine-tuning in a zero-shot fashion.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 8 Oct 2023.
Builds on
33 of 48 references.
References · 33
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2020Exploring Simple Siamese Representation Learningcited by 2,866
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
- 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
21 more
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
- 2023Scaling Speech Technology to 1,000+ Languagescited by 411
- 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
- 2019Billion-scale semi-supervised learning for image classificationcited by 232
- 2019pyannote.audio: neural building blocks for speaker diarizationcited by 181
- 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
- 2017The Zero Resource Speech Challenge 2017cited by 45
- 2022Word Discovery in Visually Grounded, Self-Supervised Speech Modelscited by 37
- 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
- 2021Segmental Contrastive Predictive Coding for Unsupervised Word Segmentationcited by 32
- 2018Truly unsupervised acoustic word embeddings using weak top-down constraints in encoder-decoder modelscited by 29
- 2016Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approachescited by 25
- 2022Word Segmentation on Discovered Phone Units with Dynamic Programming and Self-Supervised Scoringcited by 24
- 2022Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimationcited by 23
- 2020A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddingscited by 12
- 2022DP-Parse: Finding Word Boundaries from Raw Speech with an Instance Lexiconcited by 12
- 2022Phoneme Segmentation Using Self-Supervised Speech Modelscited by 11
- 2023Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Modelcited by 11
- 2022Speech Sequence Embeddings using Nearest Neighbors Contrastive Learningcited by 7
- 2023Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labelscited by 7
Built on by
Citing papers per year: 2024: 1.
1 paper.