# Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0 acoustic model

arXiv: https://arxiv.org/abs/2104.02558 · HTML: https://mlsanity.com/paper/2104.02558/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [wav2letter++: The Fastest Open-source Speech Recognition System](https://mlsanity.com/paper/1812.07625/) (2018)
- [Unsupervised Pre-training of Bidirectional Speech Encoders via Masked Reconstruction](https://mlsanity.com/paper/2001.10603/) (2020)
- [Espresso: A Fast End-to-end Neural Speech Recognition Toolkit](https://mlsanity.com/paper/1909.08723/) (2019)
- [PyChain: A Fully Parallelized PyTorch Implementation of LF-MMI for End-to-End ASR](https://mlsanity.com/paper/2005.09824/) (2020)
- [Parallel training of DNNs with Natural Gradient and Parameter Averaging](https://mlsanity.com/paper/1410.7455/) (2014)
- [Pkwrap: a PyTorch Package for LF-MMI Training of Acoustic Models](https://mlsanity.com/paper/2010.03466/) (2020)
- [Lattice-Free MMI Adaptation Of Self-Supervised Pretrained Acoustic Models](https://mlsanity.com/paper/2012.14252/) (2020)
- [Improving LSTM-CTC based ASR performance in domains with limited training data](https://mlsanity.com/paper/1707.00722/) (2017)

## Built on by (most cited first, 6 of 6)

- [How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications](https://mlsanity.com/paper/2203.16822/) (2022)
- [Boosting Cross-Domain Speech Recognition with Self-Supervision](https://mlsanity.com/paper/2206.09783/) (2022)
- [Are disentangled representations all you need to build speaker anonymization systems?](https://mlsanity.com/paper/2208.10497/) (2022)
- [DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model](https://mlsanity.com/paper/2303.09278/) (2023)
- [Improving Low-Resource Speech Recognition with Pretrained Speech Models: Continued Pretraining vs. Semi-Supervised Training](https://mlsanity.com/paper/2207.00659/) (2022)
- [Effectiveness of Text, Acoustic, and Lattice-based representations in Spoken Language Understanding tasks](https://mlsanity.com/paper/2212.08489/) (2022)

## Abstract (arXiv)

In this work, we investigate if the wav2vec 2.0 self-supervised pretraining helps mitigate the overfitting issues with connectionist temporal classification (CTC) training to reduce its performance gap with flat-start lattice-free MMI (E2E-LFMMI) for automatic speech recognition with limited training data. Towards that objective, we use the pretrained wav2vec 2.0 BASE model and fine-tune it on three different datasets including out-of-domain (Switchboard) and cross-lingual (Babel) scenarios. Our results show that for supervised adaptation of the wav2vec 2.0 model, both E2E-LFMMI and CTC achieve similar results; significantly outperforming the baselines trained only with supervised data. Fine-tuning the wav2vec 2.0 model with E2E-LFMMI and CTC we obtain the following relative WER improvements over the supervised baseline trained with E2E-LFMMI. We get relative improvements of 40% and 44% on the clean-set and 64% and 58% on the test set of Librispeech (100h) respectively. On Switchboard (300h) we obtain relative improvements of 33% and 35% respectively. Finally, for Babel languages, we obtain relative improvements of 26% and 23% on Swahili (38h) and 18% and 17% on Tagalog (84h) respectively.