mlsanityData to 7 Oct 2026

Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision

arXiv 2401.00273 ↗First posted 30 Dec 2023eess.AS cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

This work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found that self-supervised models can achieve performances close to the supervised model, indicating the effectiveness of multilingual self-supervised pre-training. We also observed that these models still have room for improvement as they kept making similar mistakes and had unsatisfactory performances on modeling intra-sentential code-switching. In addition, the validity of several variants of Whisper was explored, and we concluded that they remained effective in a code-switching scenario, and similar techniques for self-supervised models are worth studying to boost the performance of code-switched tasks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 20 of its 23 references that are in the map. First posted on arXiv 30 Dec 2023.

Builds on

20 of 23 references.

References · 20

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2023Scaling Speech Technology to 1,000+ Languagescited by 401
  6. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  7. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  8. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
  9. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  10. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  11. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 99
  12. 2021ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversationcited by 43
8 more
  1. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  2. 2023Can Whisper perform speech-based in-context learning?cited by 35
  3. 2022Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarizationcited by 28
  4. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  5. 2023Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuningcited by 13
  6. 2023Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languagescited by 12
  7. 2023Adapting the adapters for code-switching in multilingual ASRcited by 6
  8. 2022SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scalecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  2. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  3. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 38
  4. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  5. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  6. 2024Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whispercited by 12
  7. 2025A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Datacited by 7
  8. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Datacited by 5
  9. 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
  10. 2024Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modelingcited by 4
  11. 2025Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semanticallycited by 3
  12. 2026ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Modelscited by 3
1 more
  1. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignmentcited by 1