DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model
Not on the map. Why?
Abstract (arXiv)
Multilingual self-supervised speech representation models have greatly enhanced the speech recognition performance for low-resource languages, and the compression of these huge models has also become a crucial prerequisite for their industrial application. In this paper, we propose DistilXLSR, a distilled cross-lingual speech representation model. By randomly shuffling the phonemes of existing speech, we reduce the linguistic information and distill cross-lingual models using only English data. We also design a layer-jumping initialization method to fully leverage the teacher's pre-trained weights. Experiments on 2 kinds of teacher models and 15 low-resource languages show that our method can reduce the parameters by 50% while maintaining cross-lingual representation ability. Our method is proven to be generalizable to various languages/teacher models and has the potential to improve the cross-lingual performance of the English pre-trained models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 2 Jun 2023.
Builds on
18 of 24 references.
References · 18
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
- 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
- 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
- 2019Structured Pruning of Large Language Modelscited by 201
- 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
- 2021Simple and Effective Zero-shot Cross-lingual Phoneme Recognitioncited by 79
6 more
- 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
- 2020Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languagescited by 30
- 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
- 2021On Addressing Practical Challenges for RNN-Transducercited by 19
- 2022Knowledge Distillation of Transformer-based Language Models Revisitedcited by 11
- 2022MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformerscited by 5
Built on by
Citing papers per year: 2023: 1.
1 paper.