# Large-scale Transfer Learning for Low-resource Spoken Language Understanding

arXiv: https://arxiv.org/abs/2008.05671 · HTML: https://mlsanity.com/paper/2008.05671/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2020.

## Built on by (most cited first, 11 of 11)

- [Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systems](https://mlsanity.com/paper/2203.00006/) (2022)
- [On the Use of External Data for Spoken Named Entity Recognition](https://mlsanity.com/paper/2112.07648/) (2021)
- [End-to-end model for named entity recognition from speech without paired training data](https://mlsanity.com/paper/2204.00803/) (2022)
- [Integrating Dialog History into End-to-End Spoken Language Understanding Systems](https://mlsanity.com/paper/2108.08405/) (2021)
- [RNN Transducer Models For Spoken Language Understanding](https://mlsanity.com/paper/2104.03842/) (2021)
- [Contextual Text Embeddings for Twi](https://mlsanity.com/paper/2103.15963/) (2021)
- [Speech Augmentation Based Unsupervised Learning for Keyword Spotting](https://mlsanity.com/paper/2205.14329/) (2022)
- [Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approach](https://mlsanity.com/paper/2210.13805/) (2022)
- [A Study of Different Ways to Use The Conformer Model For Spoken Language Understanding](https://mlsanity.com/paper/2204.03879/) (2022)
- [Adaptive Activation Network For Low Resource Multilingual Speech Recognition](https://mlsanity.com/paper/2205.14326/) (2022)
- [Meta Auxiliary Learning for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.12774/) (2022)

## Abstract (arXiv)

End-to-end Spoken Language Understanding (SLU) models are made increasingly large and complex to achieve the state-ofthe-art accuracy. However, the increased complexity of a model can also introduce high risk of over-fitting, which is a major challenge in SLU tasks due to the limitation of available data. In this paper, we propose an attention-based SLU model together with three encoder enhancement strategies to overcome data sparsity challenge. The first strategy focuses on the transferlearning approach to improve feature extraction capability of the encoder. It is implemented by pre-training the encoder component with a quantity of Automatic Speech Recognition annotated data relying on the standard Transformer architecture and then fine-tuning the SLU model with a small amount of target labelled data. The second strategy adopts multitask learning strategy, the SLU model integrates the speech recognition model by sharing the same underlying encoder, such that improving robustness and generalization ability. The third strategy, learning from Component Fusion (CF) idea, involves a Bidirectional Encoder Representation from Transformer (BERT) model and aims to boost the capability of the decoder with an auxiliary network. It hence reduces the risk of over-fitting and augments the ability of the underlying encoder, indirectly. Experiments on the FluentAI dataset show that cross-language transfer learning and multi-task strategies have been improved by up to 4:52% and 3:89% respectively, compared to the baseline.