# Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

arXiv: https://arxiv.org/abs/2401.02921 · HTML: https://mlsanity.com/paper/2401.02921/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension](https://mlsanity.com/paper/1804.00320/) (2018)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)
- [Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understanding](https://mlsanity.com/paper/2305.13512/) (2023)
- [SpeechGen: Unlocking the Generative Power of Speech Language Models with Prompts](https://mlsanity.com/paper/2306.02207/) (2023)
- [SpeechPrompt: An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks](https://mlsanity.com/paper/2203.16773/) (2022)
- [DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering](https://mlsanity.com/paper/2203.04911/) (2022)
- [Adapting Pretrained Transformer to Lattices for Spoken Language Understanding](https://mlsanity.com/paper/2011.00780/) (2020)
- [N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses](https://mlsanity.com/paper/2106.06519/) (2021)
- [Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understanding](https://mlsanity.com/paper/2005.11640/) (2020)

## Built on by (most cited first, 8 of 8)

- [Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations](https://mlsanity.com/paper/2402.12786/) (2024)
- [ASR Error Correction using Large Language Models](https://mlsanity.com/paper/2409.09554/) (2024)
- [Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbation](https://mlsanity.com/paper/2407.10303/) (2024)
- [MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues](https://mlsanity.com/paper/2408.14418/) (2024)
- [Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLM](https://mlsanity.com/paper/2506.11089/) (2025)
- [FlanEC: Exploring Flan-T5 for Post-ASR Error Correction](https://mlsanity.com/paper/2501.12979/) (2025)
- [Proactive for Uncertainty: Cause-Aware Error Diagnosis and Interactive Clarification for Spoken Dialogue Systems](https://mlsanity.com/paper/2605.25404/) (2026)
- [CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents](https://mlsanity.com/paper/2606.21453/) (2026)

## Abstract (arXiv)

In the realm of spoken language understanding (SLU), numerous natural language understanding (NLU) methodologies have been adapted by supplying large language models (LLMs) with transcribed speech instead of conventional written text. In real-world scenarios, prior to input into an LLM, an automated speech recognition (ASR) system generates an output transcript hypothesis, where inherent errors can degrade subsequent SLU tasks. Here we introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis, aiming to encapsulate speech ambiguities and enhance SLU outcomes. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts with the help of word confusion networks from lattices, bridging the SLU performance gap between using the top ASR hypothesis and an oracle upper bound. Additionally, we delve into the LLM's robustness to varying ASR performance conditions and scrutinize the aspects of in-context learning which prove the most influential.