Generalized zero-shot audio-to-intent classification
Not on the map. Why?
Abstract (arXiv)
Spoken language understanding systems using audio-only data are gaining popularity, yet their ability to handle unseen intents remains limited. In this study, we propose a generalized zero-shot audio-to-intent classification framework with only a few sample text sentences per intent. To achieve this, we first train a supervised audio-to-intent classifier by making use of a self-supervised pre-trained model. We then leverage a neural audio synthesizer to create audio embeddings for sample text utterances and perform generalized zero-shot classification on unseen intents using cosine similarity. We also propose a multimodal training strategy that incorporates lexical information into the audio representation to improve zero-shot performance. Our multimodal training approach improves the accuracy of zero-shot intent classification on unseen intents of SLURP by 2.75% and 18.2% for the SLURP and internal goal-oriented dialog datasets, respectively, compared to audio-only training.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 4 Nov 2023.
Builds on
23 of 38 references.
References · 23
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
- 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2017Feature Generating Networks for Zero-Shot Learningcited by 257
- 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
- 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
- 2021Contrastive Embedding for Generalized Zero-Shot Learningcited by 94
- 2018Multi-modal Cycle-consistent Generalized Zero-Shot Learningcited by 87
11 more
- 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
- 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
- 2022Audio Retrieval with WavText5K and CLAP Trainingcited by 35
- 2020Zero-Shot Audio Classification via Semantic Embeddingscited by 21
- 2021Generalized Zero-shot Intent Detection via Commonsense Knowledgecited by 21
- 2021Representation learning through cross-modal conditional teacher-student training for speech emotion recognitioncited by 17
- 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
- 2020Transformer based unsupervised pre-training for acoustic representation learningcited by 9
- 2021Listen with Intent: Improving Speech Recognition with Audio-to-Intent Front-Endcited by 8
- 2022An Analysis of Semantically-Aligned Speech-Text Embeddingscited by 5
- 2022Improved Zero-Shot Audio Tagging & Classification with Patchout Spectrogram Transformerscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.