# Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languages

arXiv: https://arxiv.org/abs/2310.03018 · HTML: https://mlsanity.com/paper/2310.03018/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling](https://mlsanity.com/paper/2011.11588/) (2020)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarization](https://mlsanity.com/paper/2210.14567/) (2022)
- [End-to-End Speech Translation for Code Switched Speech](https://mlsanity.com/paper/2204.05076/) (2022)
- [Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptation](https://mlsanity.com/paper/2203.16104/) (2022)
- [Code-Switching without Switching: Language Agnostic End-to-End Speech Translation](https://mlsanity.com/paper/2210.01512/) (2022)
- [Towards Natural Bilingual and Code-Switched Speech Synthesis Based on Mix of Monolingual Recordings and Cross-Lingual Voice Conversion](https://mlsanity.com/paper/2010.08136/) (2020)
- [Ensemble knowledge distillation of self-supervised speech models](https://mlsanity.com/paper/2302.12757/) (2023)
- [Improving generalizability of distilled self-supervised speech processing models under distorted settings](https://mlsanity.com/paper/2210.07978/) (2022)
- [Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition](https://mlsanity.com/paper/2211.16319/) (2022)

## Built on by (most cited first, 12 of 12)

- [DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment](https://mlsanity.com/paper/2507.02768/) (2025)
- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation](https://mlsanity.com/paper/2407.09886/) (2024)
- [AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Models](https://mlsanity.com/paper/2506.05140/) (2025)
- [Investigating Zero-Shot Generalizability on Mandarin-English Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models with Self-Supervision and Weak Supervision](https://mlsanity.com/paper/2401.00273/) (2023)
- [Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper](https://mlsanity.com/paper/2406.05806/) (2024)
- [A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data](https://mlsanity.com/paper/2506.11130/) (2025)
- [Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data](https://mlsanity.com/paper/2407.10603/) (2024)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [SAGE: Spliced-Audio Generated Data for Enhancing Foundational Models in Low-Resource Arabic-English Code-Switched Speech Recognition](https://mlsanity.com/paper/2506.22143/) (2025)
- [MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models](https://mlsanity.com/paper/2603.09714/) (2026)
- [Multilingual Information Retrieval with a Monolingual Knowledge Base](https://mlsanity.com/paper/2506.02527/) (2025)

## Abstract (arXiv)

We introduce a new zero resource code-switched speech benchmark designed to directly assess the code-switching capabilities of self-supervised speech encoders. We showcase a baseline system of language modeling on discrete units to demonstrate how the code-switching abilities of speech encoders can be assessed in a zero-resource manner. Our experiments encompass a variety of well-known speech encoders, including Wav2vec 2.0, HuBERT, XLSR, etc. We examine the impact of pre-training languages and model size on benchmark performance. Notably, though our results demonstrate that speech encoders with multilingual pre-training, exemplified by XLSR, outperform monolingual variants (Wav2vec 2.0, HuBERT) in code-switching scenarios, there is still substantial room for improvement in their code-switching linguistic abilities.