mlsanityData to 14 Oct 2026

Adapting Whisper for Parameter-efficient Code-Switching Speech Recognition via Soft Prompt Tuning

arXiv 2506.21576 ↗First posted 16 Jun 2025cs.CL cs.AI cs.SDcited by 5

Not on the map. Why?

Abstract (arXiv)

Large-scale multilingual ASR models like Whisper excel in high-resource settings but face challenges in low-resource scenarios, such as rare languages and code-switching (CS), due to computational costs and catastrophic forgetting. We explore Soft Prompt Tuning (SPT), a parameter-efficient method to enhance CS ASR while preserving prior knowledge. We evaluate two strategies: (1) full fine-tuning (FFT) of both soft prompts and the entire Whisper model, demonstrating improved cross-lingual capabilities compared to traditional methods, and (2) adhering to SPT's original design by freezing model parameters and only training soft prompts. Additionally, we introduce SPT4ASR, a combination of different SPT variants. Experiments on the SEAME and ASRU2019 datasets show that deep prompt tuning is the most effective SPT approach, and our SPT4ASR methods achieve further error reductions in CS ASR, maintaining parameter efficiency similar to LoRA, without degrading performance on existing languages.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 29 of its 31 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

29 of 31 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  8. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  9. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  10. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  11. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  12. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 36
17 more
  1. 2023Can Whisper perform speech-based in-context learning?cited by 35
  2. 2022Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarizationcited by 29
  3. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
  4. 2023Adaptation of Whisper models to child speech recognitioncited by 26
  5. 2022Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Promptcited by 25
  6. 2022LAE: Language-Aware Encoder for Monolingual and Multilingual ASRcited by 22
  7. 2023N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognitioncited by 20
  8. 2021Joint Modeling of Code-Switched and Monolingual ASR via Conditional Factorizationcited by 18
  9. 2024Learn and Don't Forget: Adding a New Language to ASR Foundation Modelscited by 16
  10. 2023Extending Whisper with prompt tuning to target-speaker ASRcited by 14
  11. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  12. 2021Minimum word error training for non-autoregressive Transformer-based code-switching ASRcited by 9
  13. 2024Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whispercited by 9
  14. 2023Adapting OpenAI's Whisper for Speech Recognition on Code-Switch Mandarin-English SEAME and ASRU2019 Datasetscited by 8
  15. 2024Soft Prompt Tuning for Cross-Lingual Transfer: When Less is Morecited by 7
  16. 2022Internal Language Model Estimation based Language Model Fusion for Cross-Domain Code-Switching Speech Recognitioncited by 5
  17. 2023Are Soft Prompts Good Zero-shot Learners for Speech Recognition?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Improving Code-Switching ASR with Code-Mixing Guided Synthetic Speechcited by 1
  2. 2026Adding Robust Code-Switching Capabilities to High Performance Multilingual ASRcited by 0
  3. 2026BuzzASR: A Swarm of 100+ Monolingual Speech Recognition Modelscited by 0
  4. 2026Beyond WER: Entity and Disfluency Recall in Accented Conversational ASRcited by 0
  5. 2026NADI 2026: The Second Multidialectal Arabic Speech Processing Shared Taskcited by 0