# RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition

arXiv: https://arxiv.org/abs/1805.05225 · HTML: https://mlsanity.com/paper/1805.05225/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2018.

## Built on by (most cited first, 30 of 30)

- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Why does CTC result in peaky behavior?](https://mlsanity.com/paper/2105.14849/) (2021)
- [Librispeech Transducer Model with Internal Language Model Prior Correction](https://mlsanity.com/paper/2104.03006/) (2021)
- [Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models](https://mlsanity.com/paper/2104.05544/) (2021)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Improving the Training Recipe for a Robust Conformer-based Hybrid Model](https://mlsanity.com/paper/2206.12955/) (2022)
- [VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain](https://mlsanity.com/paper/2404.05659/) (2024)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder](https://mlsanity.com/paper/2409.14074/) (2024)
- [A study of latent monotonic attention variants](https://mlsanity.com/paper/2103.16710/) (2021)
- [Monotonic segmental attention for automatic speech recognition](https://mlsanity.com/paper/2210.14742/) (2022)
- [Mixture Encoder for Joint Speech Separation and Recognition](https://mlsanity.com/paper/2306.12173/) (2023)
- [Comparative Analysis of the wav2vec 2.0 Feature Extractor](https://mlsanity.com/paper/2308.04286/) (2023)
- [AppTek's Submission to the IWSLT 2022 Isometric Spoken Language Translation Task](https://mlsanity.com/paper/2205.05807/) (2022)
- [RASR2: The RWTH ASR Toolkit for Generic Sequence-to-sequence Speech Recognition](https://mlsanity.com/paper/2305.17782/) (2023)
- [Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text](https://mlsanity.com/paper/2306.03557/) (2023)
- [The Conformer Encoder May Reverse the Time Dimension](https://mlsanity.com/paper/2410.00680/) (2024)
- [Regularizing Learnable Feature Extraction for Automatic Speech Recognition](https://mlsanity.com/paper/2506.09804/) (2025)
- [Supplementary Resources and Analysis for Automatic Speech Recognition Systems Trained on the Loquacious Dataset](https://mlsanity.com/paper/2512.17915/) (2025)
- [On Sampling-Based Training Criteria for Neural Language Modeling](https://mlsanity.com/paper/2104.10507/) (2021)
- [Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2306.07949/) (2023)
- [Developing automatic verbatim transcripts for international multilingual meetings: an end-to-end solution](https://mlsanity.com/paper/2309.15609/) (2023)
- [Weak Alignment Supervision from Hybrid Model Improves End-to-end ASR](https://mlsanity.com/paper/2311.14835/) (2023)
- [wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech](https://mlsanity.com/paper/2408.04174/) (2024)
- [Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression](https://mlsanity.com/paper/2501.18895/) (2025)
- [Dynamic Acoustic Model Architecture Optimization in Training for ASR](https://mlsanity.com/paper/2506.13180/) (2025)
- [Automatic Learning of Subword Dependent Model Scales](https://mlsanity.com/paper/2110.09324/) (2021)
- [Analyzing And Improving Neural Speaker Embeddings for ASR](https://mlsanity.com/paper/2301.04571/) (2023)
- [Alternating Weak Triphone/BPE Alignment Supervision from Hybrid Model Improves End-to-End ASR](https://mlsanity.com/paper/2402.15594/) (2024)

## Abstract (arXiv)

We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure TensorFlow beam search decoder. We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks. Promising preliminary results on max. expected BLEU training are presented. We are able to train state-of-the-art models for translation and end-to-end models for speech recognition and show results on WMT 2017 and Switchboard. The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications.