# Three-Module Modeling For End-to-End Spoken Language Understanding Using Pre-trained DNN-HMM-Based Acoustic-Phonetic Model

arXiv: https://arxiv.org/abs/2204.03315 · HTML: https://mlsanity.com/paper/2204.03315/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Speech Model Pre-training for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/1904.03670/) (2019)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Transformer-based Online CTC/attention End-to-End Speech Recognition Architecture](https://mlsanity.com/paper/2001.08290/) (2020)
- [Exploring Transfer Learning For End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2012.08549/) (2020)

## Built on by (most cited first, 3 of 3)

- [Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.14318/) (2022)
- [A Study of Different Ways to Use The Conformer Model For Spoken Language Understanding](https://mlsanity.com/paper/2204.03879/) (2022)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)

## Abstract (arXiv)

In spoken language understanding (SLU), what the user says is converted to his/her intent. Recent work on end-to-end SLU has shown that accuracy can be improved via pre-training approaches. We revisit ideas presented by Lugosch et al. using speech pre-training and three-module modeling; however, to ease construction of the end-to-end SLU model, we use as our phoneme module an open-source acoustic-phonetic model from a DNN-HMM hybrid automatic speech recognition (ASR) system instead of training one from scratch. Hence we fine-tune on speech only for the word module, and we apply multi-target learning (MTL) on the word and intent modules to jointly optimize SLU performance. MTL yields a relative reduction of 40% in intent-classification error rates (from 1.0% to 0.6%). Note that our three-module model is a streaming method. The final outcome of the proposed three-module modeling approach yields an intent accuracy of 99.4% on FluentSpeech, an intent error rate reduction of 50% compared to that of Lugosch et al. Although we focus on real-time streaming methods, we also list non-streaming methods for comparison.