# Advancing CTC-CRF Based End-to-End Speech Recognition with Wordpieces and Conformers

arXiv: https://arxiv.org/abs/2107.03007 · HTML: https://mlsanity.com/paper/2107.03007/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [Jasper: An End-to-End Convolutional Neural Acoustic Model](https://mlsanity.com/paper/1904.03288/) (2019)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [SpecAugment on Large Scale Datasets](https://mlsanity.com/paper/1912.05533/) (2019)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [From Senones to Chenones: Tied Context-Dependent Graphemes for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.01493/) (2019)
- [Phoneme Based Neural Transducer for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/2010.16368/) (2020)
- [Multistream CNN for Robust Acoustic Modeling](https://mlsanity.com/paper/2005.10470/) (2020)
- [CAT: A CTC-CRF based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches towards Data Efficiency and Low Latency](https://mlsanity.com/paper/2005.13326/) (2020)
- [A Parallelizable Lattice Rescoring Strategy with Neural Language Models](https://mlsanity.com/paper/2103.05081/) (2021)
- [Faster, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces](https://mlsanity.com/paper/2005.09150/) (2020)
- [Efficient Neural Architecture Search for End-to-end Speech Recognition via Straight-Through Gradients](https://mlsanity.com/paper/2011.05649/) (2020)
- [Deformable TDNN with adaptive receptive fields for speech recognition](https://mlsanity.com/paper/2104.14791/) (2021)

## Built on by (most cited first, 4 of 4)

- [CTC Variations Through New WFST Topologies](https://mlsanity.com/paper/2110.03098/) (2021)
- [Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision](https://mlsanity.com/paper/2406.02166/) (2024)
- [Global Normalization for Streaming Speech Recognition in a Modular Framework](https://mlsanity.com/paper/2205.13674/) (2022)
- [Low-Resourced Speech Recognition for Iu Mien Language via Weakly-Supervised Phoneme-based Multilingual Pre-training](https://mlsanity.com/paper/2407.13292/) (2024)

## Abstract (arXiv)

Automatic speech recognition systems have been largely improved in the past few decades and current systems are mainly hybrid-based and end-to-end-based. The recently proposed CTC-CRF framework inherits the data-efficiency of the hybrid approach and the simplicity of the end-to-end approach. In this paper, we further advance CTC-CRF based ASR technique with explorations on modeling units and neural architectures. Specifically, we investigate techniques to enable the recently developed wordpiece modeling units and Conformer neural networks to be succesfully applied in CTC-CRFs. Experiments are conducted on two English datasets (Switchboard, Librispeech) and a German dataset from CommonVoice. Experimental results suggest that (i) Conformer can improve the recognition performance significantly; (ii) Wordpiece-based systems perform slightly worse compared with phone-based systems for the target language with a low degree of grapheme-phoneme correspondence (e.g. English), while the two systems can perform equally strong when such degree of correspondence is high for the target language (e.g. German).