Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding
Not on the map. Why?
Abstract (arXiv)
End-to-end (E2E) spoken language understanding (SLU) systems that generate a semantic parse from speech have become more promising recently. This approach uses a single model that utilizes audio and text representations from pre-trained speech recognition models (ASR), and outperforms traditional pipeline SLU systems in on-device streaming scenarios. However, E2E SLU systems still show weakness when text representation quality is low due to ASR transcription errors. To overcome this issue, we propose a novel E2E SLU system that enhances robustness to ASR errors by fusing audio and text representations based on the estimated modality confidence of ASR hypotheses. We introduce two novel techniques: 1) an effective method to encode the quality of ASR hypotheses and 2) an effective approach to integrate them into E2E SLU models. We show accuracy improvements on STOP dataset and share the analysis to demonstrate the effectiveness of our approach.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 22 Jul 2023.
Builds on
30 of 34 references.
References · 30
- 2017Attention Is All You Needcited by 59,932
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
- 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
- 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
- 2018Towards end-to-end spoken language understandingcited by 92
- 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
- 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
- 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
- 2019Two-Pass End-to-End Speech Recognitioncited by 62
18 more
- 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
- 2021Interspeech 2021 Deep Noise Suppression Challengecited by 43
- 2020Alignment Restricted Streaming Recurrent Neural Network Transducercited by 42
- 2020Conversational Semantic Parsingcited by 39
- 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
- 2020Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfacescited by 24
- 2020Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducercited by 22
- 2022STOP: A dataset for Spoken Task Oriented Semantic Parsingcited by 22
- 2020Parallel Rescoring with Transformer for Streaming On-Device Speech Recognitioncited by 9
- 2021A Streaming End-to-End Framework For Spoken Language Understandingcited by 8
- 2021FANS: Fusing ASR and NLU for on-device SLUcited by 8
- 2022Deliberation Model for On-Device Spoken Language Understandingcited by 8
- 2022Two-Pass Low Latency End-to-End Spoken Language Understandingcited by 8
- 2021Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speechcited by 6
- 2021Streaming Transformer Transducer Based Speech Recognition Using Non-Causal Convolutioncited by 5
- 2022Joint Audio/Text Training for Transformer Rescorer of Streaming Speech Recognitioncited by 5
- 2022End-to-End Spoken Language Understanding: Performance analyses of a voice command task in a low resource settingcited by 3
- 2022Introducing Semantics into Speech Encoderscited by 3
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.
2 papers.