# End-to-End Attention-based Large Vocabulary Speech Recognition

arXiv: https://arxiv.org/abs/1508.04395 · HTML: https://mlsanity.com/paper/1508.04395/

## Key facts

- Cited by 151 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2015.

## Built on by (most cited first, 30 of 151)

- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions](https://mlsanity.com/paper/2104.02724/) (2021)
- [UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units](https://mlsanity.com/paper/2212.08055/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection](https://mlsanity.com/paper/2201.12806/) (2022)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [Auxiliary Losses for Learning Generalizable Concept-based Models](https://mlsanity.com/paper/2311.11108/) (2023)
- [Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer](https://mlsanity.com/paper/2105.02412/) (2021)
- [Tree-constrained Pointer Generator for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2109.00627/) (2021)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [End-to-End Speech Recognition from Federated Acoustic Models](https://mlsanity.com/paper/2104.14297/) (2021)
- [Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation](https://mlsanity.com/paper/2104.02775/) (2021)
- [AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations](https://mlsanity.com/paper/2302.06419/) (2023)
- [Pre-training for Speech Translation: CTC Meets Optimal Transport](https://mlsanity.com/paper/2301.11716/) (2023)
- [Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning](https://mlsanity.com/paper/2112.03603/) (2021)
- [An evaluation of word-level confidence estimation for end-to-end automatic speech recognition](https://mlsanity.com/paper/2101.05525/) (2021)
- [End-to-End Spoken Language Understanding for Generalized Voice Assistants](https://mlsanity.com/paper/2106.09009/) (2021)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [Accelerating RNN-T Training and Inference Using CTC guidance](https://mlsanity.com/paper/2210.16481/) (2022)
- [Attention layers provably solve single-location regression](https://mlsanity.com/paper/2410.01537/) (2024)
- [Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities](https://mlsanity.com/paper/2211.05756/) (2022)
- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [An End-to-End Multi-Task Learning Model for Image-based Table Recognition](https://mlsanity.com/paper/2303.08648/) (2023)
- [Domain-aware Neural Language Models for Speech Recognition](https://mlsanity.com/paper/2101.03229/) (2021)

## Abstract (arXiv)

Many of the current state-of-the-art Large Vocabulary Continuous Speech Recognition Systems (LVCSR) are hybrids of neural networks and Hidden Markov Models (HMMs). Most of these systems contain separate components that deal with the acoustic modelling, language modelling and sequence decoding. We investigate a more direct approach in which the HMM is replaced with a Recurrent Neural Network (RNN) that performs sequence prediction directly at the character level. Alignment between the input features and the desired character sequence is learned automatically by an attention mechanism built into the RNN. For each predicted character, the attention mechanism scans the input sequence and chooses relevant frames. We propose two methods to speed up this operation: limiting the scan to a subset of most promising frames and pooling over time the information contained in neighboring frames, thereby reducing source sequence length. Integrating an n-gram language model into the decoding process yields recognition accuracies similar to other HMM-free RNN-based approaches.