# Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learning

arXiv: https://arxiv.org/abs/2104.10357 · HTML: https://mlsanity.com/paper/2104.10357/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 21 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Speech-language Pre-training for End-to-end Spoken Language Understanding](https://mlsanity.com/paper/2102.06283/) (2021)
- [Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces](https://mlsanity.com/paper/2008.06173/) (2020)
- [Learning ASR-Robust Contextualized Embeddings for Spoken Language Understanding](https://mlsanity.com/paper/1909.10861/) (2019)
- [Robust Neural Machine Translation with Joint Textual and Phonetic Embedding](https://mlsanity.com/paper/1810.06729/) (2018)
- [Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation](https://mlsanity.com/paper/1906.01199/) (2019)
- [Recent Advances in End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1909.13332/) (2019)
- [Speech to Text Adaptation: Towards an Efficient Cross-Modal Distillation](https://mlsanity.com/paper/2005.08213/) (2020)
- [Phone Features Improve Speech Translation](https://mlsanity.com/paper/2005.13681/) (2020)
- [Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification](https://mlsanity.com/paper/2102.07370/) (2021)
- [Spoken Language Intent Detection using Confusion2Vec](https://mlsanity.com/paper/1904.03576/) (2019)
- [Improving the Robustness of Speech Translation](https://mlsanity.com/paper/1811.00728/) (2018)
- [Learning Spoken Language Representations with Neural Lattice Language Modeling](https://mlsanity.com/paper/2007.02629/) (2020)

## Built on by (most cited first, 6 of 6)

- [Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.14318/) (2022)
- [PWESuite: Phonetic Word Embeddings and Tasks They Facilitate](https://mlsanity.com/paper/2304.02541/) (2023)
- [Investigation of Japanese PnG BERT language model in text-to-speech synthesis for pitch accent language](https://mlsanity.com/paper/2212.08321/) (2022)
- [Calibrate and Refine! A Novel and Agile Framework for ASR-error Robust Intent Detection](https://mlsanity.com/paper/2205.11008/) (2022)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)
- [Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding](https://mlsanity.com/paper/2405.15097/) (2024)

## Abstract (arXiv)

In the traditional cascading architecture for spoken language understanding (SLU), it has been observed that automatic speech recognition errors could be detrimental to the performance of natural language understanding. End-to-end (E2E) SLU models have been proposed to directly map speech input to desired semantic frame with a single model, hence mitigating ASR error propagation. Recently, pre-training technologies have been explored for these E2E models. In this paper, we propose a novel joint textual-phonetic pre-training approach for learning spoken language representations, aiming at exploring the full potentials of phonetic information to improve SLU robustness to ASR errors. We explore phoneme labels as high-level speech features, and design and compare pre-training tasks based on conditional masked language model objectives and inter-sentence relation objectives. We also investigate the efficacy of combining textual and phonetic information during fine-tuning. Experimental results on spoken language understanding benchmarks, Fluent Speech Commands and SNIPS, show that the proposed approach significantly outperforms strong baseline models and improves robustness of spoken language understanding to ASR errors.