# Building Robust Spoken Language Understanding by Cross Attention between Phoneme Sequence and ASR Hypothesis

arXiv: https://arxiv.org/abs/2203.12067 · HTML: https://mlsanity.com/paper/2203.12067/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 23 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript](https://mlsanity.com/paper/2102.00804/) (2021)
- [ST-BERT: Cross-modal Language Model Pre-training For End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2010.12283/) (2020)
- [Improving Spoken Language Understanding By Exploiting ASR N-best Hypotheses](https://mlsanity.com/paper/2001.05284/) (2020)
- [Spoken Language Intent Detection using Confusion2Vec](https://mlsanity.com/paper/1904.03576/) (2019)
- [Learning Spoken Language Representations with Neural Lattice Language Modeling](https://mlsanity.com/paper/2007.02629/) (2020)

## Abstract (arXiv)

Building Spoken Language Understanding (SLU) robust to Automatic Speech Recognition (ASR) errors is an essential issue for various voice-enabled virtual assistants. Considering that most ASR errors are caused by phonetic confusion between similar-sounding expressions, intuitively, leveraging the phoneme sequence of speech can complement ASR hypothesis and enhance the robustness of SLU. This paper proposes a novel model with Cross Attention for SLU (denoted as CASLU). The cross attention block is devised to catch the fine-grained interactions between phoneme and word embeddings in order to make the joint representations catch the phonetic and semantic features of input simultaneously and for overcoming the ASR errors in downstream natural language understanding (NLU) tasks. Extensive experiments are conducted on three datasets, showing the effectiveness and competitiveness of our approach. Additionally, We also validate the universality of CASLU and prove its complementarity when combining with other robust SLU techniques.