# Insertion-Based Modeling for End-to-End Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2005.13211 · HTML: https://mlsanity.com/paper/2005.13211/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 13 of 13)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation](https://mlsanity.com/paper/2110.05249/) (2021)
- [Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models](https://mlsanity.com/paper/2107.09428/) (2021)
- [BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model](https://mlsanity.com/paper/2210.16663/) (2022)
- [An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2106.09885/) (2021)
- [TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition](https://mlsanity.com/paper/2104.01522/) (2021)
- [Non-autoregressive Transformer with Unified Bidirectional Decoder for Automatic Speech Recognition](https://mlsanity.com/paper/2109.06684/) (2021)
- [InterAug: Augmenting Noisy Intermediate Predictions for CTC-based ASR](https://mlsanity.com/paper/2204.00174/) (2022)
- [Sim-T: Simplify the Transformer Network by Multiplexing Technique for Speech Recognition](https://mlsanity.com/paper/2304.04991/) (2023)
- [GNCformer Enhanced Self-attention for Automatic Speech Recognition](https://mlsanity.com/paper/2305.12755/) (2023)
- [Improving CTC-AED model with integrated-CTC and auxiliary loss regularization](https://mlsanity.com/paper/2308.08449/) (2023)

## Abstract (arXiv)

End-to-end (E2E) models have gained attention in the research field of automatic speech recognition (ASR). Many E2E models proposed so far assume left-to-right autoregressive generation of an output token sequence except for connectionist temporal classification (CTC) and its variants. However, left-to-right decoding cannot consider the future output context, and it is not always optimal for ASR. One of the non-left-to-right models is known as non-autoregressive Transformer (NAT) and has been intensively investigated in the area of neural machine translation (NMT) research. One NAT model, mask-predict, has been applied to ASR but the model needs some heuristics or additional component to estimate the length of the output token sequence. This paper proposes to apply another type of NAT called insertion-based models, that were originally proposed for NMT, to ASR tasks. Insertion-based models solve the above mask-predict issues and can generate an arbitrary generation order of an output sequence. In addition, we introduce a new formulation of joint training of the insertion-based models and CTC. This formulation reinforces CTC by making it dependent on insertion-based token generation in a non-autoregressive manner. We conducted experiments on three public benchmarks and achieved competitive performance to strong autoregressive Transformer with a similar decoding condition.