mlsanityData to 7 Oct 2026

From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs

arXiv 2603.24034 ↗First posted 25 Mar 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on https://github.com/XYGuo1996/Contextual_Speech_LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 46 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

31 of 46 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  8. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  9. 2015Listen, Attend and Spellcited by 542
  10. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  11. 2020Contextual RNN-T For Open Domain ASRcited by 85
  12. 2025Step-Audio: Unified Understanding and Generation in Intelligent Speech Interactioncited by 83
19 more
  1. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  2. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  3. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
  4. 2024Improving ASR Contextual Biasing with Guided Attentioncited by 17
  5. 2024Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervisioncited by 17
  6. 2023SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Abilitycited by 16
  7. 2024Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbationcited by 11
  8. 2024LLMs Are Zero-Shot Context-Aware Simultaneous Translatorscited by 10
  9. 2024Contextualization of ASR with LLM using phonetic retrieval-based augmentationcited by 10
  10. 2025HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignmentcited by 10
  11. 2024Contextualized Automatic Speech Recognition with Dynamic Vocabularycited by 8
  12. 2020Improving accuracy of rare words for RNN-Transducer through unigram shallow fusioncited by 6
  13. 2025Hearing More with Less: Multi-Modal Retrieval-and-Selection Augmented Conversational LLM-Based ASRcited by 4
  14. 2023Enhancing End-to-End Conversational Speech Translation Through Target Language Context Utilizationcited by 3
  15. 2024Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learningcited by 2
  16. 2024Deep CLAS: Deep Contextual Listen, Attend and Spellcited by 2
  17. 2025Retrieval-Augmented Speech Recognition Approach for Domain Challengescited by 2
  18. 2025Joint decoding method for controllable contextual speech recognition based on Speech LLMcited by 2
  19. 2024Self-consistent context aware conformer transducer for speech recognitioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning
  2. 2026When Does Context Help? Error Dynamics of Contextual Information in Large Language Models
  3. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  4. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
  5. 2025END: Early Noise Dropping for Efficient and Effective Context Denoising
  6. 2025Reproducing and Dissecting Denoising Language Models for Speech Recognition
  7. 2025Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandings
  8. 2025Conversational Speech Reveals Structural Robustness Failures in SpeechLLM Backbones