# Iterative Shallow Fusion of Backward Language Model for End-to-End Speech Recognition

arXiv: https://arxiv.org/abs/2310.11010 · HTML: https://mlsanity.com/paper/2310.11010/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Cold Fusion: Training Seq2Seq Models Together with Language Models](https://mlsanity.com/paper/1708.06426/) (2017)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition](https://mlsanity.com/paper/2002.11268/) (2020)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Achieving Human Parity in Conversational Speech Recognition](https://mlsanity.com/paper/1610.05256/) (2016)
- [Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM](https://mlsanity.com/paper/1706.02737/) (2017)
- [English Conversational Telephone Speech Recognition by Humans and Machines](https://mlsanity.com/paper/1703.02136/) (2017)
- [Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboard](https://mlsanity.com/paper/2001.07263/) (2020)
- [On the limit of English conversational speech recognition](https://mlsanity.com/paper/2105.00982/) (2021)
- [The JHU Multi-Microphone Multi-Speaker ASR System for the CHiME-6 Challenge](https://mlsanity.com/paper/2006.07898/) (2020)
- [Back from the future: bidirectional CTC decoding using future information in speech recognition](https://mlsanity.com/paper/2110.03326/) (2021)

## Built on by (most cited first, 1 of 1)

- [SMILE: Speech Meta In-Context Learning for Low-Resource Language Automatic Speech Recognition](https://mlsanity.com/paper/2409.10429/) (2024)

## Abstract (arXiv)

We propose a new shallow fusion (SF) method to exploit an external backward language model (BLM) for end-to-end automatic speech recognition (ASR). The BLM has complementary characteristics with a forward language model (FLM), and the effectiveness of their combination has been confirmed by rescoring ASR hypotheses as post-processing. In the proposed SF, we iteratively apply the BLM to partial ASR hypotheses in the backward direction (i.e., from the possible next token to the start symbol) during decoding, substituting the newly calculated BLM scores for the scores calculated at the last iteration. To enhance the effectiveness of this iterative SF (ISF), we train a partial sentence-aware BLM (PBLM) using reversed text data including partial sentences, considering the framework of ISF. In experiments using an attention-based encoder-decoder ASR system, we confirmed that ISF using the PBLM shows comparable performance with SF using the FLM. By performing ISF, early pruning of prospective hypotheses can be prevented during decoding, and we can obtain a performance improvement compared to applying the PBLM as post-processing. Finally, we confirmed that, by combining SF and ISF, further performance improvement can be obtained thanks to the complementarity of the FLM and PBLM.