# Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition

arXiv: https://arxiv.org/abs/2301.00066 · HTML: https://mlsanity.com/paper/2301.00066/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 30 Dec 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018)
- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [Adapt-and-Adjust: Overcoming the Long-Tail Problem of Multilingual Speech Recognition](https://mlsanity.com/paper/2012.01687/) (2020)
- [Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus](https://mlsanity.com/paper/2008.10491/) (2020)
- [The NLP Task Effectiveness of Long-Range Transformers](https://mlsanity.com/paper/2202.07856/) (2022)
- [Multi-task Language Modeling for Improving Speech Recognition of Rare Words](https://mlsanity.com/paper/2011.11715/) (2020)
- [Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation](https://mlsanity.com/paper/2205.01546/) (2022)
- [Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech Recognition](https://mlsanity.com/paper/2203.05008/) (2022)
- [Lookup-Table Recurrent Language Models for Long Tail Speech Recognition](https://mlsanity.com/paper/2104.04552/) (2021)
- [Improving Proper Noun Recognition in End-to-End ASR By Customization of the MWER Loss Criterion](https://mlsanity.com/paper/2005.09756/) (2020)

## Built on by (most cited first, 1 of 1)

- [Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models](https://mlsanity.com/paper/2601.07372/) (2026)

## Abstract (arXiv)

Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens that appear less frequently in the training set is still quite challenging. The long-tail prediction problems have been widely studied in many applications, but only been addressed by a few studies for ASR and LMs. In this paper, we propose a new memory augmented lookup dictionary based Transformer architecture for LM. The newly introduced lookup dictionary incorporates rich contextual information in training set, which is vital to correctly predict long-tail tokens. With intensive experiments on Chinese and English data sets, our proposed method is proved to outperform the baseline Transformer LM by a great margin on both word/character error rate and tail tokens error rate. This is achieved without impact on the decoding efficiency. Overall, we demonstrate the effectiveness of our proposed method in boosting the ASR decoding performance, especially for long-tail tokens.