# Streaming end-to-end speech recognition with jointly trained neural feature enhancement

arXiv: https://arxiv.org/abs/2105.01254 · HTML: https://mlsanity.com/paper/2105.01254/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017)
- [Attention based on-device streaming speech recognition with large speech corpus](https://mlsanity.com/paper/2001.00577/) (2020)
- [Efficient Implementation of the Room Simulator for Training Deep Neural Network Acoustic Models](https://mlsanity.com/paper/1712.03439/) (2017)
- [end-to-end training of a large vocabulary end-to-end speech recognition system](https://mlsanity.com/paper/1912.11040/) (2019)
- [power-law nonlinearity with maximally uniform distribution criterion for improved neural network training in automatic speech recognition](https://mlsanity.com/paper/1912.11041/) (2019)

## Built on by (most cited first, 4 of 4)

- [Speech enhancement with frequency domain auto-regressive modeling](https://mlsanity.com/paper/2309.13537/) (2023)
- [Data-driven grapheme-to-phoneme representations for a lexicon-free text-to-speech](https://mlsanity.com/paper/2401.10465/) (2024)
- [Audio Enhancement for Computer Audition -- An Iterative Training Paradigm Using Sample Importance](https://mlsanity.com/paper/2408.06264/) (2024)
- [Macro-block dropout for improved regularization in training end-to-end speech recognition models](https://mlsanity.com/paper/2212.14149/) (2022)

## Abstract (arXiv)

In this paper, we present a streaming end-to-end speech recognition model based on Monotonic Chunkwise Attention (MoCha) jointly trained with enhancement layers. Even though the MoCha attention enables streaming speech recognition with recognition accuracy comparable to a full attention-based approach, training this model is sensitive to various factors such as the difficulty of training examples, hyper-parameters, and so on. Because of these issues, speech recognition accuracy of a MoCha-based model for clean speech drops significantly when a multi-style training approach is applied. Inspired by Curriculum Learning [1], we introduce two training strategies: Gradual Application of Enhanced Features (GAEF) and Gradual Reduction of Enhanced Loss (GREL). With GAEF, the model is initially trained using clean features. Subsequently, the portion of outputs from the enhancement layers gradually increases. With GREL, the portion of the Mean Squared Error (MSE) loss for the enhanced output gradually reduces as training proceeds. In experimental results on the LibriSpeech corpus and noisy far-field test sets, the proposed model with GAEF-GREL training strategies shows significantly better results than the conventional multi-style training approach.