Are you speaking my languages? On spoken language adherence in multimodal LLMs
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
While Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. We formally define this challenge as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: (1) zero-shot prompting for robust guidance under uncertainty, (2) supervised fine-tuning (SFT) to improve prompt adherence, and (3) Chain-of-Thought (CoT) reasoning to enforce adherence during decoding. We present a comparative analysis of these methods across multiple languages, evaluating effectiveness in reducing the language violation while maintaining overall ASR performance. Finally, we discuss trade-offs to guide strategy selection under various compute constraints.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 9 of its 21 references that are in the map. First posted on arXiv 15 Jun 2026.
Builds on
9 of 21 references.
References · 9
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
- 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
- 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 77
- 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
- 2024Automata-based constraints for language model decodingcited by 51
- 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
- 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
- 2022Streaming End-to-End Multilingual Speech Recognition with Joint Language Identificationcited by 24
- 2020Language-agnostic Multilingual Modelingcited by 17
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
- 2026LLM-HB: Language-Aware LLM-Guided Hotword Biasing for Code-Switching ASR
- 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
- 2023Multi-stage Large Language Model Correction for Speech Recognition
- 2023Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Models
- 2024Efficient Compression of Multitask Multilingual Speech Models
- 2026Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs