# Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq

arXiv: https://arxiv.org/abs/1805.10387 · HTML: https://mlsanity.com/paper/1805.10387/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2018.

## Built on by (most cited first, 11 of 11)

- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [8-bit Numerical Formats for Deep Neural Networks](https://mlsanity.com/paper/2206.02915/) (2022)
- [Training and inference of large language models using 8-bit floating point](https://mlsanity.com/paper/2309.17224/) (2023)
- [Unit Scaling: Out-of-the-Box Low-Precision Training](https://mlsanity.com/paper/2303.11257/) (2023)
- [Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech](https://mlsanity.com/paper/2103.01032/) (2021)
- [Collage: Light-Weight Low-Precision Strategy for LLM Training](https://mlsanity.com/paper/2405.03637/) (2024)
- [ApproxTrain: Fast Simulation of Approximate Multipliers for DNN Training and Inference](https://mlsanity.com/paper/2209.04161/) (2022)
- [Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition](https://mlsanity.com/paper/2211.15075/) (2022)
- [Evaluating OpenAI's Whisper ASR for Punctuation Prediction and Topic Modeling of life histories of the Museum of the Person](https://mlsanity.com/paper/2305.14580/) (2023)
- [Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models](https://mlsanity.com/paper/2111.03664/) (2021)
- [Accelerating Bangla NLP Tasks with Automatic Mixed Precision: Resource-Efficient Training Preserving Model Efficacy](https://mlsanity.com/paper/2512.00829/) (2025)

## Abstract (arXiv)

We present OpenSeq2Seq - a TensorFlow-based toolkit for training sequence-to-sequence models that features distributed and mixed-precision training. Benchmarks on machine translation and speech recognition tasks show that models built using OpenSeq2Seq give state-of-the-art performance at 1.5-3x less training time. OpenSeq2Seq currently provides building blocks for models that solve a wide range of tasks including neural machine translation, automatic speech recognition, and speech synthesis.