# MTLM: Incorporating Bidirectional Text Information to Enhance Language Model Training in Speech Recognition Systems

arXiv: https://arxiv.org/abs/2502.10058 · HTML: https://mlsanity.com/paper/2502.10058/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 26 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Towards better decoding and language model integration in sequence to sequence models](https://mlsanity.com/paper/1612.02695/) (2016)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Multi-task Learning based Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2012.14631/) (2020)
- [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020)
- [Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems](https://mlsanity.com/paper/2204.00212/) (2022)
- [Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition](https://mlsanity.com/paper/2108.07789/) (2021)
- [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021)
- [Transformer with Bidirectional Decoder for Speech Recognition](https://mlsanity.com/paper/2008.04481/) (2020)

## Abstract (arXiv)

Automatic speech recognition (ASR) systems normally consist of an acoustic model (AM) and a language model (LM). The acoustic model estimates the probability distribution of text given the input speech, while the language model calibrates this distribution toward a specific knowledge domain to produce the final transcription. Traditional ASR-specific LMs are typically trained in a unidirectional (left-to-right) manner to align with autoregressive decoding. However, this restricts the model from leveraging the right-side context during training, limiting its representational capacity. In this work, we propose MTLM, a novel training paradigm that unifies unidirectional and bidirectional manners through 3 training objectives: ULM, BMLM, and UMLM. This approach enhances the LM's ability to capture richer linguistic patterns from both left and right contexts while preserving compatibility with standard ASR autoregressive decoding methods. As a result, the MTLM model not only enhances the ASR system's performance but also support multiple decoding strategies, including shallow fusion, unidirectional/bidirectional n-best rescoring. Experiments on the LibriSpeech dataset show that MTLM consistently outperforms unidirectional training across multiple decoding strategies, highlighting its effectiveness and flexibility in ASR applications.