# Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM

arXiv: https://arxiv.org/abs/1706.02737 · HTML: https://mlsanity.com/paper/1706.02737/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2017.

## Built on by (most cited first, 30 of 43)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition](https://mlsanity.com/paper/2307.05956/) (2023)
- [Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers](https://mlsanity.com/paper/2104.09426/) (2021)
- [FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning](https://mlsanity.com/paper/2207.00555/) (2022)
- [Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Tasks](https://mlsanity.com/paper/2105.00573/) (2021)
- [Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech](https://mlsanity.com/paper/2108.12881/) (2021)
- [Hierarchical Transformer-based Large-Context End-to-end ASR with Large-Context Knowledge Distillation](https://mlsanity.com/paper/2102.07935/) (2021)
- [SubSpectral Normalization for Neural Audio Data Processing](https://mlsanity.com/paper/2103.13620/) (2021)
- [WNARS: WFST based Non-autoregressive Streaming End-to-End Speech Recognition](https://mlsanity.com/paper/2104.03587/) (2021)
- [Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models](https://mlsanity.com/paper/2409.18654/) (2024)
- [On Compressing Sequences for Self-Supervised Speech Models](https://mlsanity.com/paper/2210.07189/) (2022)
- [Contextualized Automatic Speech Recognition with Dynamic Vocabulary](https://mlsanity.com/paper/2405.13344/) (2024)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [A Comparative Study on Neural Architectures and Training Methods for Japanese Speech Recognition](https://mlsanity.com/paper/2106.05111/) (2021)
- [Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2102.07259/) (2021)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)
- [Joint Speech Recognition and Audio Captioning](https://mlsanity.com/paper/2202.01405/) (2022)
- [3M: Multi-loss, Multi-path and Multi-level Neural Networks for speech recognition](https://mlsanity.com/paper/2204.03178/) (2022)
- [Delayed Fusion: Integrating Large Language Models into First-Pass Decoding in End-to-end Speech Recognition](https://mlsanity.com/paper/2501.09258/) (2025)
- [Self-Attention Channel Combinator Frontend for End-to-End Multichannel Far-field Speech Recognition](https://mlsanity.com/paper/2109.04783/) (2021)
- [Sequence Transduction with Graph-based Supervision](https://mlsanity.com/paper/2111.01272/) (2021)
- [Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correction](https://mlsanity.com/paper/2408.16180/) (2024)
- [Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation](https://mlsanity.com/paper/2103.09903/) (2021)
- [Pre-training for low resource speech-to-intent applications](https://mlsanity.com/paper/2103.16674/) (2021)
- [Capturing Multi-Resolution Context by Dilated Self-Attention](https://mlsanity.com/paper/2104.02858/) (2021)
- [Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild](https://mlsanity.com/paper/2107.09510/) (2021)
- [Decoupling Visual-Semantic Feature Learning for Robust Scene Text Recognition](https://mlsanity.com/paper/2111.12351/) (2021)
- [On Comparison of Encoders for Attention based End to End Speech Recognition in Standalone and Rescoring Mode](https://mlsanity.com/paper/2206.12829/) (2022)

## Abstract (arXiv)

We present a state-of-the-art end-to-end Automatic Speech Recognition (ASR) model. We learn to listen and write characters with a joint Connectionist Temporal Classification (CTC) and attention-based encoder-decoder network. The encoder is a deep Convolutional Neural Network (CNN) based on the VGG network. The CTC network sits on top of the encoder and is jointly trained with the attention-based decoder. During the beam search process, we combine the CTC predictions, the attention-based decoder predictions and a separately trained LSTM language model. We achieve a 5-10% error reduction compared to prior systems on spontaneous Japanese and Chinese speech, and our end-to-end model beats out traditional hybrid ASR systems.