mlsanityData to 14 Oct 2026

Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech Model

arXiv 2310.02971 ↗First posted 4 Oct 2023eess.AS cs.CL eess.SPcited by 3

Not on the map. Why?

Abstract (arXiv)

Prompting and adapter tuning have emerged as efficient alternatives to fine-tuning (FT) methods. However, existing studies on speech prompting focused on classification tasks and failed on more complex sequence generation tasks. Besides, adapter tuning is primarily applied with a focus on encoder-only self-supervised models. Our experiments show that prompting on Wav2Seq, a self-supervised encoder-decoder model, surpasses previous works in sequence generation tasks. It achieves a remarkable 53% relative improvement in word error rate for ASR and a 27% in F1 score for slot filling. Additionally, prompting competes with the FT method in the low-resource scenario. Moreover, we show the transferability of prompting and adapter tuning on Wav2Seq in cross-lingual ASR. When limited trainable parameters are involved, prompting and adapter tuning consistently outperform conventional FT across 7 languages. Notably, in the low-resource scenario, prompting consistently outperforms adapter tuning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

38 of 45 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  9. 2019Language Models as Knowledge Bases?cited by 1,936
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
26 more
  1. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  2. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  3. 2017Learning multiple visual domains with residual adapterscited by 557
  4. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  5. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  6. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  7. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  8. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  9. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  10. 2021Text-Free Prosody-Aware Generative Spoken Language Modelingcited by 119
  11. 2018Adversarial Reprogramming of Neural Networkscited by 116
  12. 2023Textually Pretrained Speech Language Modelscited by 101
  13. 2022Model Reprogramming: Resource-Efficient Cross-Domain Machine Learningcited by 64
  14. 2021Lightweight Adapter Tuning for Multilingual Speech Translationcited by 54
  15. 2020Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretrainingcited by 35
  16. 2023SpeechPrompt v2: Prompt Tuning for Speech Classification Taskscited by 32
  17. 2022AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Taskscited by 29
  18. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  19. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  20. 2023From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognitioncited by 23
  21. 2023SpeechGen: Unlocking the Generative Power of Speech Language Models with Promptscited by 21
  22. 2022Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogrammingcited by 17
  23. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  24. 2022Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptationcited by 10
  25. 2022Parameter Efficient Transfer Learning for Various Speech Processing Taskscited by 9
  26. 2022Joint Encoder-Decoder Self-Supervised Pre-training for ASRcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024SpeechPrompt: Prompting Speech Language Models for Speech Processing Taskscited by 12
  2. 2023Exploring In-Context Learning of Textless Speech Language Model for Speech Classification Taskscited by 5
  3. 2024ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Fillercited by 5