# Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard

arXiv: https://arxiv.org/abs/2001.07263 · HTML: https://mlsanity.com/paper/2001.07263/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jan 2020.

## Built on by (most cited first, 30 of 36)

- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding](https://mlsanity.com/paper/2207.02971/) (2022)
- [Robust Self-Supervised Audio-Visual Speech Recognition](https://mlsanity.com/paper/2201.01763/) (2022)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Models](https://mlsanity.com/paper/2104.05544/) (2021)
- [Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers](https://mlsanity.com/paper/2104.09426/) (2021)
- [Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems](https://mlsanity.com/paper/2204.00212/) (2022)
- [On Language Model Integration for RNN Transducer based Speech Recognition](https://mlsanity.com/paper/2110.06841/) (2021)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [Efficient Training of Neural Transducer for Speech Recognition](https://mlsanity.com/paper/2204.10586/) (2022)
- [Acoustic Data-Driven Subword Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/2104.09106/) (2021)
- [Residual Energy-Based Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2103.14152/) (2021)
- [LegoNN: Building Modular Encoder-Decoder Models](https://mlsanity.com/paper/2206.03318/) (2022)
- [Improving the Training Recipe for a Robust Conformer-based Hybrid Model](https://mlsanity.com/paper/2206.12955/) (2022)
- [Comparing the Benefit of Synthetic Training Data for Various Automatic Speech Recognition Architectures](https://mlsanity.com/paper/2104.05379/) (2021)
- [4-bit Quantization of LSTM-based Speech Recognition Models](https://mlsanity.com/paper/2108.12074/) (2021)
- [Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognition](https://mlsanity.com/paper/2309.08436/) (2023)
- [A study of latent monotonic attention variants](https://mlsanity.com/paper/2103.16710/) (2021)
- [Equivalence of Segmental and Neural Transducer Modeling: A Proof of Concept](https://mlsanity.com/paper/2104.06104/) (2021)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)
- [Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition](https://mlsanity.com/paper/2112.11438/) (2021)
- [Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognition](https://mlsanity.com/paper/2109.08744/) (2021)
- [Two-pass Decoding and Cross-adaptation Based System Combination of End-to-end Conformer and Hybrid TDNN ASR Systems](https://mlsanity.com/paper/2206.11596/) (2022)
- [Monotonic segmental attention for automatic speech recognition](https://mlsanity.com/paper/2210.14742/) (2022)
- [Overcoming Domain Mismatch in Low Resource Sequence-to-Sequence ASR Models using Hybrid Generated Pseudotranscripts](https://mlsanity.com/paper/2106.07716/) (2021)
- [Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducers](https://mlsanity.com/paper/2212.04325/) (2022)
- [Reducing Exposure Bias in Training Recurrent Neural Network Transducers](https://mlsanity.com/paper/2108.10803/) (2021)
- [Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systems](https://mlsanity.com/paper/2204.05188/) (2022)
- [Accelerating Inference and Language Model Fusion of Recurrent Neural Network Transducers via End-to-End 4-bit Quantization](https://mlsanity.com/paper/2206.07882/) (2022)

## Abstract (arXiv)

It is generally believed that direct sequence-to-sequence (seq2seq) speech recognition models are competitive with hybrid models only when a large amount of data, at least a thousand hours, is available for training. In this paper, we show that state-of-the-art recognition performance can be achieved on the Switchboard-300 database using a single headed attention, LSTM based model. Using a cross-utterance language model, our single-pass speaker independent system reaches 6.4% and 12.5% word error rate (WER) on the Switchboard and CallHome subsets of Hub5'00, without a pronunciation lexicon. While careful regularization and data augmentation are crucial in achieving this level of performance, experiments on Switchboard-2000 show that nothing is more useful than more data. Overall, the combination of various regularizations and a simple but fairly large model results in a new state of the art, 4.7% and 7.8% WER on the Switchboard and CallHome sets, using SWB-2000 without any external data resources.