Few-Shot Spoken Language Understanding via Joint Speech-Text Models
Not on the map. Why?
Abstract (arXiv)
Recent work on speech representation models jointly pre-trained with text has demonstrated the potential of improving speech representations by encoding speech and text in a shared space. In this paper, we leverage such shared representations to address the persistent challenge of limited data availability in spoken language understanding tasks. By employing a pre-trained speech-text model, we find that models fine-tuned on text can be effectively transferred to speech testing data. With as little as 1 hour of labeled speech data, our proposed approach achieves comparable performance on spoken language understanding tasks (specifically, sentiment analysis and named entity recognition) when compared to previous methods using speech-only pre-trained models fine-tuned on 10 times more data. Beyond the proof-of-concept study, we also analyze the latent representations. We find that the bottom layers of speech-text models are largely task-agnostic and align speech and text representations into a shared space, while the top layers are more task-specific.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 43 of its 46 references that are in the map. First posted on arXiv 9 Oct 2023.
Builds on
43 of 46 references.
References · 43
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2019Cross-lingual Language Model Pretrainingcited by 1,090
- 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 752
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2019How multilingual is Multilingual BERT?cited by 645
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
- 2021Generative Spoken Language Modeling from Raw Audiocited by 340
- 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
- 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
- 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
31 more
- 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 257
- 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
- 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
- 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 180
- 2023Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modelingcited by 167
- 2021Direct speech-to-speech translation with discrete unitscited by 149
- 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
- 2021Textless Speech-to-Speech Translation on Real Datacited by 121
- 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
- 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
- 2020Unsupervised pretraining transfers well across languagescited by 101
- 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
- 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
- 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
- 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
- 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
- 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 65
- 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
- 2023What Do Self-Supervised Speech Models Know About Words?cited by 64
- 2022SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Datacited by 49
- 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
- 2023Wave to Syntax: Probing spoken language models for syntaxcited by 18
- 2022Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speechcited by 15
- 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
- 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
- 2022End-to-end model for named entity recognition from speech without paired training datacited by 11
- 2022Cross-lingual Similarity of Multilingual Representations Revisitedcited by 8
- 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7
- 2023Zero-Shot End-to-End Spoken Language Understanding via Cross-Modal Selective Self-Trainingcited by 5
- 2022token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Textcited by 3
- 2022Multitask Learning for Low Resource Spoken Language Understandingcited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.