mlsanityData to 14 Oct 2026

Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard

arXiv 2001.07263 ↗First posted 20 Jan 2020eess.AS cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

It is generally believed that direct sequence-to-sequence (seq2seq) speech recognition models are competitive with hybrid models only when a large amount of data, at least a thousand hours, is available for training. In this paper, we show that state-of-the-art recognition performance can be achieved on the Switchboard-300 database using a single headed attention, LSTM based model. Using a cross-utterance language model, our single-pass speaker independent system reaches 6.4% and 12.5% word error rate (WER) on the Switchboard and CallHome subsets of Hub5'00, without a pronunciation lexicon. While careful regularization and data augmentation are crucial in achieving this level of performance, experiments on Switchboard-2000 show that nothing is more useful than more data. Overall, the combination of various regularizations and a simple but fairly large model results in a new state of the art, 4.7% and 7.8% WER on the Switchboard and CallHome sets, using SWB-2000 without any external data resources.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 18; 2022: 12; 2023: 5; 2024: 0; 2025: 1.

36 papers.

Sort:

  1. 2023End-to-End Speech Recognition: A Surveycited by 129
  2. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  3. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  4. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  5. 2021On the limit of English conversational speech recognitioncited by 30
  6. 2021Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Modelscited by 25
  7. 2021Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformerscited by 25
  8. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  9. 2021On Language Model Integration for RNN Transducer based Speech Recognitioncited by 17
  10. 2021Conformer-based Hybrid ASR System for Switchboard Datasetcited by 15
  11. 2022Efficient Training of Neural Transducer for Speech Recognitioncited by 14
  12. 2021Acoustic Data-Driven Subword Modeling for End-to-End Speech Recognitioncited by 11
24 more
  1. 2021Residual Energy-Based Models for End-to-End Speech Recognitioncited by 9
  2. 2022LegoNN: Building Modular Encoder-Decoder Modelscited by 9
  3. 2022Improving the Training Recipe for a Robust Conformer-based Hybrid Modelcited by 9
  4. 2021Comparing the Benefit of Synthetic Training Data for Various Automatic Speech Recognition Architecturescited by 8
  5. 20214-bit Quantization of LSTM-based Speech Recognition Modelscited by 8
  6. 2023Chunked Attention-based Encoder-Decoder Model for Streaming Speech Recognitioncited by 7
  7. 2021A study of latent monotonic attention variantscited by 5
  8. 2021Equivalence of Segmental and Neural Transducer Modeling: A Proof of Conceptcited by 5
  9. 2021Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximationcited by 5
  10. 2021Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognitioncited by 5
  11. 2021Dual-Encoder Architecture with Encoder Selection for Joint Close-Talk and Far-Talk Speech Recognitioncited by 4
  12. 2022Two-pass Decoding and Cross-adaptation Based System Combination of End-to-end Conformer and Hybrid TDNN ASR Systemscited by 4
  13. 2022Monotonic segmental attention for automatic speech recognitioncited by 4
  14. 2021Overcoming Domain Mismatch in Low Resource Sequence-to-Sequence ASR Models using Hybrid Generated Pseudotranscriptscited by 3
  15. 2022Lattice-Free Sequence Discriminative Training for Phoneme-Based Neural Transducerscited by 3
  16. 2021Reducing Exposure Bias in Training Recurrent Neural Network Transducerscited by 2
  17. 2022Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systemscited by 2
  18. 2022Accelerating Inference and Language Model Fusion of Recurrent Neural Network Transducers via End-to-End 4-bit Quantizationcited by 2
  19. 2025Label-Context-Dependent Internal Language Model Estimation for CTCcited by 2
  20. 2023On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducerscited by 1
  21. 2023Iterative Shallow Fusion of Backward Language Model for End-to-End Speech Recognitioncited by 1
  22. 2021ChannelAugment: Improving generalization of multi-channel ASR by training with input channel randomizationcited by 0
  23. 2022Improving End-to-End Models for Set Prediction in Spoken Language Understandingcited by 0
  24. 2023Investigating the Effect of Language Models in Sequence Discriminative Training for Neural Transducerscited by 0