Exploring Energy-based Language Models with Different Architectures and Training Methods for Speech Recognition
Not on the map. Why?
Abstract (arXiv)
Energy-based language models (ELMs) parameterize an unnormalized distribution for natural sentences and are radically different from popular autoregressive language models (ALMs). As an important application, ELMs have been successfully used as a means for calculating sentence scores in speech recognition, but they all use less-modern CNN or LSTM networks. The recent progress in Transformer networks and large pretrained models such as BERT and GPT2 opens new possibility to further advancing ELMs. In this paper, we explore different architectures of energy functions and different training methods to investigate the capabilities of ELMs in rescoring for speech recognition, all using large pretrained models as backbones.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 34 references that are in the map. First posted on arXiv 22 May 2023.
Builds on
21 of 34 references.
References · 21
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2015Sequence Level Training with Recurrent Neural Networkscited by 576
- 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 405
- 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
- 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
- 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
- 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 127
- 2020Residual Energy-Based Models for Text Generationcited by 95
- 2019UER: An Open-Source Toolkit for Pre-training Modelscited by 62
- 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
9 more
- 2016Globally Normalized Transition-Based Neural Networkscited by 26
- 2019Global Autoregressive Models for Data-Efficient Sequence Learningcited by 15
- 2021Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Modelscited by 14
- 2022Global Normalization for Streaming Speech Recognition in a Modular Frameworkcited by 10
- 2017Learning neural trans-dimensional random field language models with noise-contrastive estimationcited by 7
- 2021Improving Confidence Estimation on Out-of-Domain Data for End-to-End Speech Recognitioncited by 7
- 2018Improved training of neural trans-dimensional random field language models with dynamic noise-contrastive estimationcited by 5
- 2017Language modeling with Neural trans-dimensional random fieldscited by 4
- 2020Integrating Discrete and Neural Features via Mixed-feature Trans-dimensional Random Field Language Modelscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.