# EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding

arXiv: https://arxiv.org/abs/1507.08240 · HTML: https://mlsanity.com/paper/1507.08240/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2015.

## Built on by (most cited first, 30 of 73)

- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Hierarchically Gated Recurrent Neural Network for Sequence Modeling](https://mlsanity.com/paper/2311.04823/) (2023)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Why does CTC result in peaky behavior?](https://mlsanity.com/paper/2105.14849/) (2021)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages](https://mlsanity.com/paper/2205.01086/) (2022)
- [ASR2K: Speech Recognition for Around 2000 Languages without Audio](https://mlsanity.com/paper/2209.02842/) (2022)
- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [CTC Variations Through New WFST Topologies](https://mlsanity.com/paper/2110.03098/) (2021)
- [Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision](https://mlsanity.com/paper/2406.02166/) (2024)
- [Accelerating Transducers through Adjacent Token Merging](https://mlsanity.com/paper/2306.16009/) (2023)
- [Comparing Discrete and Continuous Space LLMs for Speech Recognition](https://mlsanity.com/paper/2409.00800/) (2024)
- [Decoder-only Architecture for Streaming End-to-end Speech Recognition](https://mlsanity.com/paper/2406.16107/) (2024)
- [Speech Emotion Recognition Using Quaternion Convolutional Neural Networks](https://mlsanity.com/paper/2111.00404/) (2021)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [Differentiable Allophone Graphs for Language-Universal Speech Recognition](https://mlsanity.com/paper/2107.11628/) (2021)
- [Star Temporal Classification: Sequence Classification with Partially Labeled Data](https://mlsanity.com/paper/2201.12208/) (2022)
- [Bypass Temporal Classification: Weakly Supervised Automatic Speech Recognition with Imperfect Transcripts](https://mlsanity.com/paper/2306.01031/) (2023)
- [Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency](https://mlsanity.com/paper/2104.02176/) (2021)
- [What do End-to-End Speech Models Learn about Speaker, Language and Channel Information? A Layer-wise and Neuron-level Analysis](https://mlsanity.com/paper/2107.00439/) (2021)
- [LegoNN: Building Modular Encoder-Decoder Models](https://mlsanity.com/paper/2206.03318/) (2022)
- [Streaming parallel transducer beam search with fast-slow cascaded encoders](https://mlsanity.com/paper/2203.15773/) (2022)
- [On Compressing Sequences for Self-Supervised Speech Models](https://mlsanity.com/paper/2210.07189/) (2022)
- [Blank-regularized CTC for Frame Skipping in Neural Transducer](https://mlsanity.com/paper/2305.11558/) (2023)
- [Self-distillation Regularized Connectionist Temporal Classification Loss for Text Recognition: A Simple Yet Effective Approach](https://mlsanity.com/paper/2308.08806/) (2023)
- [Learning from Flawed Data: Weakly Supervised Automatic Speech Recognition](https://mlsanity.com/paper/2309.15796/) (2023)
- [Multi-mode Transformer Transducer with Stochastic Future Context](https://mlsanity.com/paper/2106.09760/) (2021)

## Abstract (arXiv)

The performance of automatic speech recognition (ASR) has improved tremendously due to the application of deep neural networks (DNNs). Despite this progress, building a new ASR system remains a challenging task, requiring various resources, multiple training stages and significant expertise. This paper presents our Eesen framework which drastically simplifies the existing pipeline to build state-of-the-art ASR systems. Acoustic modeling in Eesen involves learning a single recurrent neural network (RNN) predicting context-independent targets (phonemes or characters). To remove the need for pre-generated frame labels, we adopt the connectionist temporal classification (CTC) objective function to infer the alignments between speech and label sequences. A distinctive feature of Eesen is a generalized decoding approach based on weighted finite-state transducers (WFSTs), which enables the efficient incorporation of lexicons and language models into CTC decoding. Experiments show that compared with the standard hybrid DNN systems, Eesen achieves comparable word error rates (WERs), while at the same time speeding up decoding significantly.