mlsanityData to 7 Oct 2026

Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs

arXiv 2605.23975 ↗First posted 13 May 2026cs.CL cs.SDcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Audio large language models (Audio LLMs) exhibit systematic failures in transcribing code-switching speech despite strong multilingual capabilities. Focusing on English-Mandarin, we identify three failure modes: language omission, translation-instead-of-transcription, and hallucination. We apply Direct Preference Optimization (DPO) to align models, constructing preference pairs in which chosen responses preserve mixed-language content while rejected responses mimic failure patterns. Training three Audio LLMs on 100K pairs (570 hours), we observe consistent behavioral shifts: models learn to preserve language composition rather than translating when prompted for transcription. This alignment yields MER reductions up to 89.6% (in-distribution) and 20.0% (out-of-distribution). Our findings suggest DPO can effectively elicit correct code-switching transcription behavior from multilingual Audio LLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 28 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

23 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024SpeechAlign: Aligning Speech Generation to Human Preferences8 Apr

Other references · 22

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  6. 2025Qwen2.5-Omni Technical Reportcited by 704
  7. 2024Qwen2-Audio Technical Reportcited by 543
  8. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  9. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  10. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  11. 2025Qwen3-Omni Technical Reportcited by 451
  12. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 200
10 more
  1. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  2. 2018On the End-to-End Solution to Mandarin-English Code-switching Speech Recognitioncited by 30
  3. 2022Reducing Language confusion for Code-switching Speech Recognition with Token-level Language Diarizationcited by 28
  4. 2023Enhancing Code-switching Speech Recognition with Interactive Language Biasescited by 20
  5. 2025CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognitioncited by 10
  6. 2023Speech collage: code-switched audio generation by collaging monolingual corporacited by 9
  7. 2024Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLMcited by 9
  8. 2024Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decodingcited by 7
  9. 2024SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASRcited by 6
  10. 2025Can we train ASR systems on Code-switch without real code-switch data? Case study for Singapore's languagescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Efficiently Adapting Spoken Language Models for the Singaporean Contextcited by 0
  2. 2026DiaWhisper-DPO: Role-Attributed Transcription of Clinical Interviews via Failure-Mined Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech
  2. 2026Direct Preference Density Alignment for Conversational Audio Equalization
  3. 2026Are you speaking my languages? On spoken language adherence in multimodal LLMs
  4. 2026OLA: Output Language Alignment in Code-Switched LLM Interactions
  5. 2025CS3-Bench: Evaluating and Enhancing Speech-to-Speech LLMs for Mandarin-English Code-Switching
  6. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
  7. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadox
  8. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs