# PIER: A Novel Metric for Evaluating What Matters in Code-Switching

arXiv: https://arxiv.org/abs/2501.09512 · HTML: https://mlsanity.com/paper/2501.09512/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 16 Jan 2025.

## Builds on (most cited first, 26 of 26)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies](https://mlsanity.com/paper/2301.01967/) (2023)
- [End-to-End Speech Translation for Code Switched Speech](https://mlsanity.com/paper/2204.05076/) (2022)
- [Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?](https://mlsanity.com/paper/2402.12025/) (2024)
- [The MGB-2 Challenge: Arabic Multi-Dialect Broadcast Media Recognition](https://mlsanity.com/paper/1609.05625/) (2016)
- [Enhancing Code-switching Speech Recognition with Interactive Language Biases](https://mlsanity.com/paper/2309.16953/) (2023)
- [Challenges of Computational Processing of Code-Switching](https://mlsanity.com/paper/1610.02213/) (2016)
- [Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation](https://mlsanity.com/paper/2405.04327/) (2024)
- [Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparation](https://mlsanity.com/paper/2310.20470/) (2023)
- [Towards Zero-Shot Code-Switched Speech Recognition](https://mlsanity.com/paper/2211.01458/) (2022)
- [Code-Switching without Switching: Language Agnostic End-to-End Speech Translation](https://mlsanity.com/paper/2210.01512/) (2022)
- [End-to-End Evaluation for Low-Latency Simultaneous Speech Translation](https://mlsanity.com/paper/2308.03415/) (2023)
- [Attention-Guided Adaptation for Code-Switching Speech Recognition](https://mlsanity.com/paper/2312.08856/) (2023)
- [An Effective Mixture-Of-Experts Approach For Code-Switching Speech Recognition Leveraging Encoder Disentanglement](https://mlsanity.com/paper/2402.17189/) (2024)
- [Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition](https://mlsanity.com/paper/2210.08992/) (2022)
- [Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition](https://mlsanity.com/paper/2309.11327/) (2023)
- [Perturbation-based QE: An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translation](https://mlsanity.com/paper/2305.07457/) (2023)
- [SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR](https://mlsanity.com/paper/2406.18021/) (2024)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition](https://mlsanity.com/paper/2211.16319/) (2022)

## Built on by (most cited first, 11 of 11)

- [Weight Factorization and Centralization for Continual Learning in Speech Recognition](https://mlsanity.com/paper/2506.16574/) (2025)
- [Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities](https://mlsanity.com/paper/2510.07037/) (2025)
- [HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition](https://mlsanity.com/paper/2509.24613/) (2025)
- [Adapting Language Balance in Code-Switching Speech](https://mlsanity.com/paper/2510.18724/) (2025)
- [Bayesian Low-Rank Factorization for Robust Model Adaptation](https://mlsanity.com/paper/2510.18723/) (2025)
- [Phrase-Localized Language-Contrastive Guidance: Training-Free Localized Accent Control for Code-Switching Text-to-Speech](https://mlsanity.com/paper/2609.01016/) (2026)
- [MUSCAT: MUltilingual, SCientific ConversATion Benchmark](https://mlsanity.com/paper/2604.15929/) (2026)
- [Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition](https://mlsanity.com/paper/2606.06985/) (2026)
- [Adding Robust Code-Switching Capabilities to High Performance Multilingual ASR](https://mlsanity.com/paper/2606.21990/) (2026)
- [AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition](https://mlsanity.com/paper/2608.26434/) (2026)
- [Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech](https://mlsanity.com/paper/2609.11786/) (2026)

## Abstract (arXiv)

Code-switching, the alternation of languages within a single discourse, presents a significant challenge for Automatic Speech Recognition. Despite the unique nature of the task, performance is commonly measured with established metrics such as Word-Error-Rate (WER). However, in this paper, we question whether these general metrics accurately assess performance on code-switching. Specifically, using both Connectionist-Temporal-Classification and Encoder-Decoder models, we show fine-tuning on non-code-switched data from both matrix and embedded language improves classical metrics on code-switching test sets, although actual code-switched words worsen (as expected). Therefore, we propose Point-of-Interest Error Rate (PIER), a variant of WER that focuses only on specific words of interest. We instantiate PIER on code-switched utterances and show that this more accurately describes the code-switching performance, showing huge room for improvement in future work. This focused evaluation allows for a more precise assessment of model performance, particularly in challenging aspects such as inter-word and intra-word code-switching.