# Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction

arXiv: https://arxiv.org/abs/2001.10603 · HTML: https://mlsanity.com/paper/2001.10603/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2020.

## Built on by (most cited first, 30 of 38)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers](https://mlsanity.com/paper/2204.09224/) (2022)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [Assessing the State of Self-Supervised Human Activity Recognition using Wearables](https://mlsanity.com/paper/2202.12938/) (2022)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Joint Unsupervised and Supervised Training for Multilingual ASR](https://mlsanity.com/paper/2111.08137/) (2021)
- [Contrastively Disentangled Sequential Variational Autoencoder](https://mlsanity.com/paper/2110.12091/) (2021)
- [Textless Speech Emotion Conversion using Discrete and Decomposed Representations](https://mlsanity.com/paper/2111.07402/) (2021)
- [RemixIT: Continual self-training of speech enhancement models via bootstrapped remixing](https://mlsanity.com/paper/2202.08862/) (2022)
- [Acoustic word embeddings for zero-resource languages using self-supervised contrastive learning and multilingual adaptation](https://mlsanity.com/paper/2103.10731/) (2021)
- [SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training](https://mlsanity.com/paper/2201.10207/) (2022)
- [Self-Supervised Learning for speech recognition with Intermediate layer supervision](https://mlsanity.com/paper/2112.08778/) (2021)
- [Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers](https://mlsanity.com/paper/2310.09925/) (2023)
- [Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data](https://mlsanity.com/paper/2203.17113/) (2022)
- [On-demand compute reduction with stochastic wav2vec 2.0](https://mlsanity.com/paper/2204.11934/) (2022)
- [What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis](https://mlsanity.com/paper/2107.00439/) (2021)
- [A Brief Overview of Unsupervised Neural Speech Representation Learning](https://mlsanity.com/paper/2203.01829/) (2022)
- [TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretraining](https://mlsanity.com/paper/2110.13492/) (2021)
- [Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing](https://mlsanity.com/paper/2211.01522/) (2022)
- [Improving speech recognition models with small samples for air traffic control systems](https://mlsanity.com/paper/2102.08015/) (2021)
- [Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0 acoustic model](https://mlsanity.com/paper/2104.02558/) (2021)
- [Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR](https://mlsanity.com/paper/2102.06816/) (2021)
- [Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL System](https://mlsanity.com/paper/2106.09963/) (2021)
- [On Scaling Contrastive Representations for Low-Resource Speech Recognition](https://mlsanity.com/paper/2102.00850/) (2021)
- [Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcription](https://mlsanity.com/paper/2112.08692/) (2021)

## Abstract (arXiv)

We propose an approach for pre-training speech representations via a masked reconstruction loss. Our pre-trained encoder networks are bidirectional and can therefore be used directly in typical bidirectional speech recognition models. The pre-trained networks can then be fine-tuned on a smaller amount of supervised data for speech recognition. Experiments with this approach on the LibriSpeech and Wall Street Journal corpora show promising results. We find that the main factors that lead to speech recognition improvements are: masking segments of sufficient width in both time and frequency, pre-training on a much larger amount of unlabeled data than the labeled data, and domain adaptation when the unlabeled and labeled data come from different domains. The gain from pre-training is additive to that of supervised data augmentation.