# Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents

arXiv: https://arxiv.org/abs/1811.05370 · HTML: https://mlsanity.com/paper/1811.05370/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2018.

## Built on by (most cited first, 8 of 8)

- [MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models](https://mlsanity.com/paper/2410.11710/) (2024)
- [Masked Conditional Random Fields for Sequence Labeling](https://mlsanity.com/paper/2103.10682/) (2021)
- [Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech](https://mlsanity.com/paper/2105.04719/) (2021)
- [CTRAN: CNN-Transformer-based Network for Natural Language Understanding](https://mlsanity.com/paper/2303.10606/) (2023)
- [Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers](https://mlsanity.com/paper/2306.13804/) (2023)
- [A survey of joint intent detection and slot-filling models in natural language understanding](https://mlsanity.com/paper/2101.08091/) (2021)
- [Comparison Knowledge Translation for Generalizable Image Classification](https://mlsanity.com/paper/2205.03633/) (2022)
- [Multi-grained Label Refinement Network with Dependency Structures for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/2209.04156/) (2022)

## Abstract (arXiv)

User interaction with voice-powered agents generates large amounts of unlabeled utterances. In this paper, we explore techniques to efficiently transfer the knowledge from these unlabeled utterances to improve model performance on Spoken Language Understanding (SLU) tasks. We use Embeddings from Language Model (ELMo) to take advantage of unlabeled data by learning contextualized word representations. Additionally, we propose ELMo-Light (ELMoL), a faster and simpler unsupervised pre-training method for SLU. Our findings suggest unsupervised pre-training on a large corpora of unlabeled utterances leads to significantly better SLU performance compared to training from scratch and it can even outperform conventional supervised transfer. Additionally, we show that the gains from unsupervised transfer techniques can be further improved by supervised transfer. The improvements are more pronounced in low resource settings and when using only 1000 labeled in-domain samples, our techniques match the performance of training from scratch on 10-15x more labeled in-domain data.