# Open Implementation and Study of BEST-RQ for Speech Processing

arXiv: https://arxiv.org/abs/2405.04296 · HTML: https://mlsanity.com/paper/2405.04296/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 18 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 14 of 14)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language](https://mlsanity.com/paper/2212.07525/) (2022)
- [Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?](https://mlsanity.com/paper/2306.00452/) (2023)
- [Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Compute](https://mlsanity.com/paper/2306.06672/) (2023)
- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)

## Built on by (most cited first, 9 of 9)

- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [Towards Early Prediction of Self-Supervised Speech Model Performance](https://mlsanity.com/paper/2501.05966/) (2025)
- [An Analysis of Linear Complexity Attention Substitutes with BEST-RQ](https://mlsanity.com/paper/2409.02596/) (2024)
- [BEST-RQ-Based Self-Supervised Learning for Whisper Domain Adaptation](https://mlsanity.com/paper/2510.24570/) (2025)
- [In-domain SSL pre-training and streaming ASR](https://mlsanity.com/paper/2509.12101/) (2025)
- [Where Do Self-Supervised Speech Models Become Unfair?](https://mlsanity.com/paper/2604.18249/) (2026)
- [Polynomial Mixing for Efficient Self-supervised Speech Encoders](https://mlsanity.com/paper/2603.00683/) (2026)
- [Streaming Speech-to-Text Translation with a SpeechLLM](https://mlsanity.com/paper/2605.14766/) (2026)
- [BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations](https://mlsanity.com/paper/2606.30700/) (2026)

## Abstract (arXiv)

Self-Supervised Learning (SSL) has proven to be useful in various speech tasks. However, these methods are generally very demanding in terms of data, memory, and computational resources. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ), is an SSL method that has shown great performance on Automatic Speech Recognition (ASR) while being simpler than other SSL methods, such as wav2vec 2.0. Despite BEST-RQ's great performance, details are lacking in the original paper, such as the amount of GPU/TPU hours used in pre-training, and there is no official easy-to-use open-source implementation. Furthermore, BEST-RQ has not been evaluated on other downstream tasks aside from ASR and speech translation. In this work, we describe a re-implementation of a Random-projection quantizer and perform a preliminary study with a comparison to wav2vec 2.0 on four downstream tasks. We discuss the details and differences of our implementation. We show that a random projection quantizer can achieve similar downstream performance as wav2vec 2.0 while decreasing training time by over a factor of two.