Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclaw
Not on the map. Why?
Abstract (arXiv)
We present a number of low-resource approaches to the tasks of the Zero Resource Speech Challenge 2021. We build on the unsupervised representations of speech proposed by the organizers as a baseline, derived from CPC and clustered with the k-means algorithm. We demonstrate that simple methods of refining those representations can narrow the gap, or even improve upon the solutions which use a high computational budget. The results lead to the conclusion that the CPC-derived representations are still too noisy for training language models, but stable enough for simpler forms of pattern matching and retrieval.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 11 of its 20 references that are in the map. First posted on arXiv 22 Jun 2021.
Builds on
11 of 20 references.
References · 11
- 2017Attention Is All You Needcited by 60,361
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
- 2018Colorless green recurrent networks dream hierarchicallycited by 213
- 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
- 2017The Zero Resource Speech Challenge 2017cited by 45
Built on by
Citing papers per year: 2021: 1; 2022: 2; 2023: 3; 2024: 1.
7 papers.
- 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
- 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challengecited by 34
- 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
- 2021Fast-Slow Transformer for Visually Grounding Speechcited by 23
- 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 22
- 2023Representation Learning With Hidden Unit Clustering For Low Resource Speech Applicationscited by 1
- 2024Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representationscited by 0
- 2024Learning Semantic Information from Raw Audio Signal Using Both Contextual and Phonetic Representations2 Feb 2024
- 2023Representation Learning With Hidden Unit Clustering For Low Resource Speech Applications14 Jul 2023
- 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering18 May 2023
- 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning17 May 2023
- 2022Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge27 Oct 2022
- 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling7 Feb 2022
- 2021Fast-Slow Transformer for Visually Grounding Speech16 Sep 2021