mlsanityData to 7 Oct 2026

Open Implementation and Study of BEST-RQ for Speech Processing

arXiv 2405.04296 ↗First posted 7 May 2024cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Self-Supervised Learning (SSL) has proven to be useful in various speech tasks. However, these methods are generally very demanding in terms of data, memory, and computational resources. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ), is an SSL method that has shown great performance on Automatic Speech Recognition (ASR) while being simpler than other SSL methods, such as wav2vec 2.0. Despite BEST-RQ's great performance, details are lacking in the original paper, such as the amount of GPU/TPU hours used in pre-training, and there is no official easy-to-use open-source implementation. Furthermore, BEST-RQ has not been evaluated on other downstream tasks aside from ASR and speech translation. In this work, we describe a re-implementation of a Random-projection quantizer and perform a preliminary study with a comparison to wav2vec 2.0 on four downstream tasks. We discuss the details and differences of our implementation. We show that a random projection quantizer can achieve similar downstream performance as wav2vec 2.0 while decreasing training time by over a factor of two.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

14 of 18 references.

References · 14

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  3. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  4. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  5. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  6. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  7. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  8. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  9. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  10. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  11. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  12. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
2 more
  1. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  2. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 4.

9 papers.

  1. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 55
  2. 2025Towards Early Prediction of Self-Supervised Speech Model Performancecited by 5
  3. 2024An Analysis of Linear Complexity Attention Substitutes with BEST-RQcited by 2
  4. 2025BEST-RQ-Based Self-Supervised Learning for Whisper Domain Adaptationcited by 2
  5. 2025In-domain SSL pre-training and streaming ASRcited by 1
  6. 2026Where Do Self-Supervised Speech Models Become Unfair?cited by 1
  7. 2026Polynomial Mixing for Efficient Self-supervised Speech Encoderscited by 0
  8. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  9. 2026BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representationscited by 0