# SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsets

arXiv: https://arxiv.org/abs/2610.19869 · HTML: https://mlsanity.com/paper/2610.19869/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 9 references that are in the map.
- First posted on arXiv 14 Oct 2026.

## Builds on (most cited first, 8 of 8)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://mlsanity.com/paper/2412.03304/) (2024)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [SEA-LION: Southeast Asian Languages in One Network](https://mlsanity.com/paper/2504.05747/) (2025)
- [SEA-HELM: Southeast Asian Holistic Evaluation of Language Models](https://mlsanity.com/paper/2502.14301/) (2025)
- [MERaLiON-AudioLLM: Bridging Audio and Language with Large Language Models](https://mlsanity.com/paper/2412.09818/) (2024)

## Abstract (arXiv)

We introduce SEA-HELM Audio, a Southeast Asian (SEA) multilingual evaluation framework that extends the text-based SEA-HELM benchmark to assess performance degradation in cascaded automatic speech recognition and large language model (ASR-LLM) pipelines. Using synthesized speech, we compare speech-input and text-input performance across five languages (Chinese, English, Malay, Indonesian, and Tamil) on multiple-choice reasoning and instruction-following tasks. We evaluate open-weight and commercial ASR-LLM systems using identical verifiers and quantify the speech gap as the difference between text and speech accuracy. Results show substantial degradation, with accuracy drops of 9.00 to 18.50 percentage points for multiple-choice reasoning and 20.00 to 42.86 points for instruction following. Tamil exhibits the largest degradation, exceeding 40 percentage points. Our findings highlight the limitations of text-only benchmarks in predicting voice-agent reliability and demonstrate the value of SEA-HELM Audio for evaluating multilingual voice-agent systems.