# Bi-APC: Bidirectional Autoregressive Predictive Coding for Unsupervised Pre-training and Its Application to Children's ASR

arXiv: https://arxiv.org/abs/2102.06816 · HTML: https://mlsanity.com/paper/2102.06816/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 15 of 15)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders](https://mlsanity.com/paper/1910.12638/) (2019)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [Effectiveness of self-supervised pre-training for speech recognition](https://mlsanity.com/paper/1911.03912/) (2019)
- [Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction](https://mlsanity.com/paper/2001.10603/) (2020)
- [Improving Transformer-based Speech Recognition Using Unsupervised Pre-training](https://mlsanity.com/paper/1910.09932/) (2019)
- [Exploring Transformers for Large-Scale Speech Recognition](https://mlsanity.com/paper/2005.09684/) (2020)
- [Transfer Learning from Adult to Children for Speech Recognition: Evaluation, Analysis and Recommendations](https://mlsanity.com/paper/1805.03322/) (2018)
- [Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks](https://mlsanity.com/paper/1910.10387/) (2019)
- [A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition](https://mlsanity.com/paper/1606.06871/) (2016)
- [PyKaldi2: Yet another speech toolkit based on Kaldi and PyTorch](https://mlsanity.com/paper/1907.05955/) (2019)
- [Exploring the Use of an Unsupervised Autoregressive Model as a Shared Encoder for Text-Dependent Speaker Verification](https://mlsanity.com/paper/2008.03615/) (2020)
- [Speaker Recognition for Children's Speech](https://mlsanity.com/paper/1609.07498/) (2016)

## Built on by (most cited first, 5 of 5)

- [Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models](https://mlsanity.com/paper/2406.10507/) (2024)
- [An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2106.09885/) (2021)
- [Low Resource German ASR with Untranscribed Data Spoken by Non-native Children -- INTERSPEECH 2021 Shared Task SPAPL System](https://mlsanity.com/paper/2106.09963/) (2021)
- [Improving Children's Speech Recognition by Fine-tuning Self-supervised Adult Speech Representations](https://mlsanity.com/paper/2211.07769/) (2022)
- [Towards Better Meta-Initialization with Task Augmentation for Kindergarten-aged Speech Recognition](https://mlsanity.com/paper/2202.12326/) (2022)

## Abstract (arXiv)

We present a bidirectional unsupervised model pre-training (UPT) method and apply it to children's automatic speech recognition (ASR). An obstacle to improving child ASR is the scarcity of child speech databases. A common approach to alleviate this problem is model pre-training using data from adult speech. Pre-training can be done using supervised (SPT) or unsupervised methods, depending on the availability of annotations. Typically, SPT performs better. In this paper, we focus on UPT to address the situations when pre-training data are unlabeled. Autoregressive predictive coding (APC), a UPT method, predicts frames from only one direction, limiting its use to uni-directional pre-training. Conventional bidirectional UPT methods, however, predict only a small portion of frames. To extend the benefits of APC to bi-directional pre-training, Bi-APC is proposed. We then use adaptation techniques to transfer knowledge learned from adult speech (using the Librispeech corpus) to child speech (OGI Kids corpus). LSTM-based hybrid systems are investigated. For the uni-LSTM structure, APC obtains similar WER improvements to SPT over the baseline. When applied to BLSTM, however, APC is not as competitive as SPT, but our proposed Bi-APC has comparable improvements to SPT.