mlsanityData to 7 Oct 2026

Residual Energy-Based Models for End-to-End Speech Recognition

arXiv 2103.14152 ↗First posted 25 Mar 2021eess.AS cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

End-to-end models with auto-regressive decoders have shown impressive results for automatic speech recognition (ASR). These models formulate the sequence-level probability as a product of the conditional probabilities of all individual tokens given their histories. However, the performance of locally normalised models can be sub-optimal because of factors such as exposure bias. Consequently, the model distribution differs from the underlying data distribution. In this paper, the residual energy-based model (R-EBM) is proposed to complement the auto-regressive ASR model to close the gap between the two distributions. Meanwhile, R-EBMs can also be regarded as utterance-level confidence estimators, which may benefit many downstream tasks. Experiments on a 100hr LibriSpeech dataset show that R-EBMs can reduce the word error rates (WERs) by 8.2%/6.7% while improving areas under precision-recall curves of confidence scores by 12.6%/28.4% on test-clean/test-other sets. Furthermore, on a state-of-the-art model using self-supervised learning (wav2vec 2.0), R-EBMs still significantly improves both the WER and confidence estimation performance.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 25 Mar 2021.

Builds on

26 of 39 references.

References · 26

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  5. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  6. 2015Attention-Based Models for Speech Recognitioncited by 400
  7. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  8. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  9. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  10. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  11. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  12. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
14 more
  1. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  2. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 99
  3. 2020Residual Energy-Based Models for Text Generationcited by 95
  4. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  5. 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
  6. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  7. 2020Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboardcited by 36
  8. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  9. 2020Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognitioncited by 34
  10. 2021Learning Word-Level Confidence For Subword End-to-End ASRcited by 22
  11. 2021An evaluation of word-level confidence estimation for end-to-end automatic speech recognitioncited by 18
  12. 2018Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networkscited by 12
  13. 2018Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimationcited by 11
  14. 2019Integrating Source-channel and Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 8

Built on by

Cited by year

Citing papers per year: 2021: 4; 2022: 1; 2023: 2; 2024: 1; 2025: 1.

9 papers.

  1. 2021Input Length Matters: Improving RNN-T and MWER Training for Long-form Telephony Speech Recognitioncited by 10
  2. 2021Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Predictioncited by 8
  3. 2021Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognitioncited by 7
  4. 2022Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognitioncited by 7
  5. 2021Word-level confidence estimation for RNN transducerscited by 3
  6. 2023Energy-Based Models For Speech Synthesiscited by 1
  7. 2024TeLeS: Temporal Lexeme Similarity Score to Estimate Confidence in End-to-End ASRcited by 1
  8. 2023Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition Systemcited by 0
  9. 2025Score-Based Training for Energy-Based TTS Modelscited by 0