# Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models

arXiv: https://arxiv.org/abs/2606.25436 · HTML: https://mlsanity.com/paper/2606.25436/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Prompting Large Language Models with Speech Recognition Abilities](https://mlsanity.com/paper/2307.11795/) (2023)
- [Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities](https://mlsanity.com/paper/2404.17790/) (2024)
- [CoVoST 2 and Massively Multilingual Speech-to-Text Translation](https://mlsanity.com/paper/2007.10310/) (2020)
- [End-to-End Speech Recognition Contextualization with Large Language Models](https://mlsanity.com/paper/2309.10917/) (2023)
- [Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks](https://mlsanity.com/paper/2410.11005/) (2024)
- [Building a Large Japanese Web Corpus for Large Language Models](https://mlsanity.com/paper/2404.17733/) (2024)
- [Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2312.03668/) (2023)
- [Dialect Transfer for Swiss German Speech Translation](https://mlsanity.com/paper/2310.09088/) (2023)
- [Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation](https://mlsanity.com/paper/2504.05898/) (2025)
- [Towards Speech Dialogue Translation Mediating Speakers of Different Languages](https://mlsanity.com/paper/2305.09210/) (2023)
- [BanglaDialecto: An End-to-End AI-Powered Regional Speech Standardization](https://mlsanity.com/paper/2411.10879/) (2024)

## Abstract (arXiv)

Dialogue systems based on large language models (LLMs) have advanced significantly in recent years. However, dialectal variation remains a major challenge, particularly for systems that process spoken input. LLM-based speech language models (SLMs), which integrate LLMs with speech processing components, show promise for spoken language tasks, yet their ability to comprehend dialects has not been sufficiently studied. Moreover, it remains unclear how the dialectal understanding of the base LLM affects SLM performance. This study investigates the dialectal robustness of both LLMs and SLMs using Japanese dialects as a test case. We define robustness as the ratio of performance on dialectal versus standard inputs, enabling fair comparisons. Our experiments show that SLM robustness correlates with that of their text-based counterparts. Furthermore, training with dialectal data and fine-tuning the speech encoder each improves robustness in SLMs.