# MUSCAT: MUltilingual, SCientific ConversATion Benchmark

arXiv: https://arxiv.org/abs/2604.15929 · HTML: https://mlsanity.com/paper/2604.15929/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Builds on (most cited first, 27 of 27)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)
- [End-to-end speaker segmentation for overlap-aware resegmentation](https://mlsanity.com/paper/2104.04045/) (2021)
- [CoVoST 2 and Massively Multilingual Speech-to-Text Translation](https://mlsanity.com/paper/2007.10310/) (2020)
- [ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets](https://mlsanity.com/paper/2406.08641/) (2024)
- [SHAS: Approaching optimal Segmentation for End-to-End Speech Translation](https://mlsanity.com/paper/2202.04774/) (2022)
- [DiPCo -- Dinner Party Corpus](https://mlsanity.com/paper/1909.13447/) (2019)
- [Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech](https://mlsanity.com/paper/2108.12881/) (2021)
- [SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset](https://mlsanity.com/paper/2506.00087/) (2025)
- [PIER: A Novel Metric for Evaluating What Matters in Code-Switching](https://mlsanity.com/paper/2501.09512/) (2025)
- [Summary on The Multilingual Conversational Speech Language Model Challenge: Datasets, Tasks, Baselines, and Methods](https://mlsanity.com/paper/2509.13785/) (2025)
- [On the Use/Misuse of the Term 'Phoneme'](https://mlsanity.com/paper/1907.11640/) (2019)
- [Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognition](https://mlsanity.com/paper/2210.08992/) (2022)
- [KIT's Multilingual Speech Translation System for IWSLT 2023](https://mlsanity.com/paper/2306.05320/) (2023)
- [The Second DISPLACE Challenge : DIarization of SPeaker and LAnguage in Conversational Environments](https://mlsanity.com/paper/2406.09494/) (2024)
- [Weight Factorization and Centralization for Continual Learning in Speech Recognition](https://mlsanity.com/paper/2506.16574/) (2025)
- [Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks](https://mlsanity.com/paper/2510.13979/) (2025)
- [KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularization](https://mlsanity.com/paper/2505.19679/) (2025)

## Abstract (arXiv)

The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages, creating the experience as though everyone were a multilingual speaker. To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching. However, there is currently no dataset benchmarking this situation. We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges. The benchmark consists of bilingual discussions on scientific papers between multiple speakers, each conversing in a different language. We provide a standard evaluation framework, beyond Word Error Rate (WER) enabling consistent comparison of ASR performance across languages. Experimental results demonstrate that the proposed dataset is still an open challenge for state-of-the-art ASR systems. The dataset is available in https://huggingface.co/datasets/goodpiku/muscat-eval. Keywords: multilingual, speech recognition, audio segmentation, speaker diarization