MUSCAT: MUltilingual, SCientific ConversATion Benchmark
Not on the map. Why?
Abstract (arXiv)
The goal of multilingual speech technology is to facilitate seamless communication between individuals speaking different languages, creating the experience as though everyone were a multilingual speaker. To create this experience, speech technology needs to address several challenges: Handling mixed multilingual input, specific vocabulary, and code-switching. However, there is currently no dataset benchmarking this situation. We propose a new benchmark to evaluate current Automatic Speech Recognition (ASR) systems, whether they are able to handle these challenges. The benchmark consists of bilingual discussions on scientific papers between multiple speakers, each conversing in a different language. We provide a standard evaluation framework, beyond Word Error Rate (WER) enabling consistent comparison of ASR performance across languages. Experimental results demonstrate that the proposed dataset is still an open challenge for state-of-the-art ASR systems. The dataset is available in https://huggingface.co/datasets/goodpiku/muscat-eval. Keywords: multilingual, speech recognition, audio segmentation, speaker diarization
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 17 Apr 2026.
Builds on
27 of 51 references.
References · 27
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
- 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
- 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
- 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
- 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
- 2021End-to-end speaker segmentation for overlap-aware resegmentationcited by 76
15 more
- 2020CoVoST 2 and Massively Multilingual Speech-to-Text Translationcited by 73
- 2022SHAS: Approaching optimal Segmentation for End-to-End Speech Translationcited by 26
- 2024ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasetscited by 26
- 2019DiPCo -- Dinner Party Corpuscited by 18
- 2021Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speechcited by 15
- 2025SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Datasetcited by 12
- 2025PIER: A Novel Metric for Evaluating What Matters in Code-Switchingcited by 11
- 2025Summary on The Multilingual Conversational Speech Language Model Challenge: Datasets, Tasks, Baselines, and Methodscited by 10
- 2019On the Use/Misuse of the Term 'Phoneme'cited by 9
- 2022Language-agnostic Code-Switching in Sequence-To-Sequence Speech Recognitioncited by 7
- 2023KIT's Multilingual Speech Translation System for IWSLT 2023cited by 7
- 2025Weight Factorization and Centralization for Continual Learning in Speech Recognitioncited by 6
- 2024The Second DISPLACE Challenge : DIarization of SPeaker and LAnguage in Conversational Environmentscited by 4
- 2025Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talkscited by 4
- 2025KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularizationcited by 3
Built on by
Not cited yet.