# Ideal-LLM: Integrating Dual Encoders and Language-Adapted LLM for Multilingual Speech-to-Text

arXiv: https://arxiv.org/abs/2409.11214 · HTML: https://mlsanity.com/paper/2409.11214/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2024.

## Built on by (most cited first, 5 of 5)

- [When Large Language Models Meet Speech: A Survey on Integration Approaches](https://mlsanity.com/paper/2502.19548/) (2025)
- [From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM](https://mlsanity.com/paper/2503.10620/) (2025)
- [Language Family Matters: Evaluating LLM-Based ASR Across Linguistic Boundaries](https://mlsanity.com/paper/2601.18899/) (2026)
- [A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenario](https://mlsanity.com/paper/2412.00721/) (2024)
- [SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge](https://mlsanity.com/paper/2507.03343/) (2025)

## Abstract (arXiv)

Integrating audio encoders with LLMs through connectors has enabled these models to process and comprehend audio modalities, significantly enhancing speech-to-text tasks, including automatic speech recognition (ASR) and automatic speech translation (AST). However, these methods often overlook the critical aspect of language adaptation in multilingual settings, relying instead on multilingual data without adequately addressing language differences. To address this gap, we propose the Ideal-LLM model, which employs dual multilingual encoders to enrich language feature information and utilizes a language-adapted connector to target the adaptation of each language specifically. By leveraging the complementary strengths of Whisper and MMS encoders, our approach ensures richer multilingual representations. Additionally, the language-adapted connector enhances modal transformation via a language weight selector tailored for each language. Experimental results demonstrate that Ideal-LLM significantly improves ASR performance, achieving a 32.6% relative reduction in average word error rates compared to the standard speech encoder integrated with LLMs and yields an average BLEU score of 36.78 for AST task.