# Transformer-based Acoustic Modeling for Hybrid Speech Recognition

arXiv: https://arxiv.org/abs/1910.09799 · HTML: https://mlsanity.com/paper/1910.09799/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2019.

## Built on by (most cited first, 30 of 88)

- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [MobTCast: Leveraging Auxiliary Trajectory Forecasting for Human Mobility Prediction](https://mlsanity.com/paper/2110.01401/) (2021)
- [CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings](https://mlsanity.com/paper/2106.03143/) (2021)
- [Learning to Estimate Shapley Values with Vision Transformers](https://mlsanity.com/paper/2206.05282/) (2022)
- [Transformer Based Deliberation for Two-Pass Speech Recognition](https://mlsanity.com/paper/2101.11577/) (2021)
- [Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition](https://mlsanity.com/paper/2106.07759/) (2021)
- [Personalized Speech Enhancement: New Models and Comprehensive Evaluation](https://mlsanity.com/paper/2110.09625/) (2021)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)
- [Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2202.10290/) (2022)
- [End-to-End Multi-Channel Transformer for Speech Recognition](https://mlsanity.com/paper/2102.03951/) (2021)
- [Scaling ASR Improves Zero and Few Shot Learning](https://mlsanity.com/paper/2111.05948/) (2021)
- [Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2205.06445/) (2022)
- [SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing](https://mlsanity.com/paper/2302.14638/) (2023)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [SepTr: Separable Transformer for Audio Spectrogram Processing](https://mlsanity.com/paper/2203.09581/) (2022)
- [Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision](https://mlsanity.com/paper/2103.04037/) (2021)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [Multi-Head State Space Model for Speech Recognition](https://mlsanity.com/paper/2305.12498/) (2023)
- [Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correction](https://mlsanity.com/paper/2306.09313/) (2023)
- [Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition](https://mlsanity.com/paper/2201.05554/) (2022)
- [Revisiting Checkpoint Averaging for Neural Machine Translation](https://mlsanity.com/paper/2210.11803/) (2022)
- [Arabic Speech Recognition by End-to-End, Modular Systems and Human](https://mlsanity.com/paper/2101.08454/) (2021)
- [HiQuE: Hierarchical Question Embedding Network for Multimodal Depression Detection](https://mlsanity.com/paper/2408.03648/) (2024)
- [Contrastive Semi-supervised Learning for ASR](https://mlsanity.com/paper/2103.05149/) (2021)
- [XLST: Cross-lingual Self-training to Learn Multilingual Representation for Low Resource Speech Recognition](https://mlsanity.com/paper/2103.08207/) (2021)

## Abstract (arXiv)

We propose and evaluate transformer-based acoustic models (AMs) for hybrid speech recognition. Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers. We also present a preliminary study of using limited right context in transformer models, which makes it possible for streaming applications. We demonstrate that on the widely used Librispeech benchmark, our transformer-based AM outperforms the best published hybrid result by 19% to 26% relative when the standard n-gram language model (LM) is used. Combined with neural network LM for rescoring, our proposed approach achieves state-of-the-art results on Librispeech. Our findings are also confirmed on a much larger internal dataset.