mlsanityData to 14 Oct 2026

Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition

arXiv 2107.01275 ↗First posted 2 Jul 2021eess.AS cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently, attention-based encoder-decoder (AED) models have shown high performance for end-to-end automatic speech recognition (ASR) across several tasks. Addressing overconfidence in such models, in this paper we introduce the concept of relaxed attention, which is a simple gradual injection of a uniform distribution to the encoder-decoder attention weights during training that is easily implemented with two lines of code. We investigate the effect of relaxed attention across different AED model architectures and two prominent ASR tasks, Wall Street Journal (WSJ) and Librispeech. We found that transformers trained with relaxed attention outperform the standard baseline models consistently during decoding with external language models. On WSJ, we set a new benchmark for transformer-based end-to-end speech recognition with a word error rate of 3.65%, outperforming state of the art (4.20%) by 13.1% relative, while introducing only a single hyperparameter.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

29 of 42 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  11. 2019When Does Label Smoothing Help?cited by 988
  12. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
17 more
  1. 2015Listen, Attend and Spellcited by 543
  2. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  3. 2015Attention-Based Models for Speech Recognitioncited by 401
  4. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  5. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  6. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  7. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  8. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  9. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  10. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  11. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
  12. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  13. 2019Espresso: A Fast End-to-end Neural Speech Recognition Toolkitcited by 31
  14. 2018End-to-end Speech Recognition with Word-based RNN Language Modelscited by 23
  15. 2020Attention based on-device streaming speech recognition with large speech corpuscited by 18
  16. 2021Multi-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognitioncited by 6
  17. 2020Focus on the present: a regularization method for the ASR source-target attention layercited by 2

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1.

4 papers.

  1. 2022Minimum Latency Training of Sequence Transducers for Streaming End-to-End Speech Recognitioncited by 8
  2. 2022Relaxed Attention for Transformer Modelscited by 5
  3. 2022Analysis of Self-Attention Head Diversity for Conformer-based Automatic Speech Recognitioncited by 3
  4. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learningcited by 1