# N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses

arXiv: https://arxiv.org/abs/2106.06519 · HTML: https://mlsanity.com/paper/2106.06519/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Adapting Pretrained Transformer to Lattices for Spoken Language Understanding](https://mlsanity.com/paper/2011.00780/) (2020)
- [Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understanding](https://mlsanity.com/paper/2005.11640/) (2020)
- [Exploiting Sentence and Context Representations in Deep Neural Models for Spoken Language Understanding](https://mlsanity.com/paper/1610.04120/) (2016)
- [A Hierarchical Decoding Model For Spoken Language Understanding From Unaligned Data](https://mlsanity.com/paper/1904.04498/) (2019)
- [Encoding Word Confusion Networks with Recurrent Neural Networks for Dialog State Tracking](https://mlsanity.com/paper/1707.05853/) (2017)

## Built on by (most cited first, 12 of 12)

- [Can Generative Large Language Models Perform ASR Error Correction?](https://mlsanity.com/paper/2307.04172/) (2023)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [Translation-Enhanced Multilingual Text-to-Image Generation](https://mlsanity.com/paper/2305.19216/) (2023)
- [Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks](https://mlsanity.com/paper/2401.02921/) (2024)
- [Automatic Speech Recognition with BERT and CTC Transformers: A Review](https://mlsanity.com/paper/2410.09456/) (2024)
- [OLISIA: a Cascade System for Spoken Dialogue State Tracking](https://mlsanity.com/paper/2304.11073/) (2023)
- [Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding](https://mlsanity.com/paper/2205.00693/) (2022)
- [End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics](https://mlsanity.com/paper/2211.03511/) (2022)
- [Personalization for BERT-based Discriminative Speech Recognition Rescoring](https://mlsanity.com/paper/2307.06832/) (2023)
- [TaskMix: Data Augmentation for Meta-Learning of Spoken Intent Understanding](https://mlsanity.com/paper/2210.06341/) (2022)
- [Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home](https://mlsanity.com/paper/2306.00482/) (2023)

## Abstract (arXiv)

Spoken Language Understanding (SLU) systems parse speech into semantic structures like dialog acts and slots. This involves the use of an Automatic Speech Recognizer (ASR) to transcribe speech into multiple text alternatives (hypotheses). Transcription errors, common in ASRs, impact downstream SLU performance negatively. Approaches to mitigate such errors involve using richer information from the ASR, either in form of N-best hypotheses or word-lattices. We hypothesize that transformer models learn better with a simpler utterance representation using the concatenation of the N-best ASR alternatives, where each alternative is separated by a special delimiter [SEP]. In our work, we test our hypothesis by using concatenated N-best ASR alternatives as the input to transformer encoder models, namely BERT and XLM-RoBERTa, and achieve performance equivalent to the prior state-of-the-art model on DSTC2 dataset. We also show that our approach significantly outperforms the prior state-of-the-art when subjected to the low data regime. Additionally, this methodology is accessible to users of third-party ASR APIs which do not provide word-lattice information.