mlsanityData to 14 Oct 2026

Customizing Speech Recognition Model with Large Language Model Feedback

arXiv 2506.11091 ↗First posted 5 Jun 2025cs.CL cs.SD eess.AScited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Automatic speech recognition (ASR) systems have achieved strong performance on general transcription tasks. However, they continue to struggle with recognizing rare named entities and adapting to domain mismatches. In contrast, large language models (LLMs), trained on massive internet-scale datasets, are often more effective across a wide range of domains. In this work, we propose a reinforcement learning based approach for unsupervised domain adaptation, leveraging unlabeled data to enhance transcription quality, particularly the named entities affected by domain mismatch, through feedback from a LLM. Given contextual information, our framework employs a LLM as the reward model to score the hypotheses from the ASR model. These scores serve as reward signals to fine-tune the ASR model via reinforcement learning. Our method achieves a 21% improvement on entity word error rate over conventional self-training methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 39 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

32 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Direct Language Model Alignment from Online AI Feedback7 Feb

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2024Phi-4 Technical Reportcited by 711
  8. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145
  9. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  10. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  11. 2023Can Generative Large Language Models Perform ASR Error Correction?cited by 57
  12. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
19 more
  1. 2020Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Trainingcited by 39
  2. 2018Toward domain-invariant speech recognition via large scale trainingcited by 35
  3. 2020Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 35
  4. 2019Conditional Teacher-Student Learningcited by 34
  5. 2017Large-Scale Domain Adaptation via Teacher-Student Learningcited by 31
  6. 2023Revisiting Large Language Models as Zero-shot Relation Extractorscited by 29
  7. 2021Large-scale ASR Domain Adaptation using Self- and Semi-supervised Learningcited by 26
  8. 2024ASR Error Correction using Large Language Modelscited by 24
  9. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  10. 2018A Multi-Discriminator CycleGAN for Unsupervised Non-Parallel Speech Domain Adaptationcited by 10
  11. 2022A Call for Clarity in Beam Search: How It Works and When It Stopscited by 10
  12. 2022Boosting Cross-Domain Speech Recognition with Self-Supervisioncited by 10
  13. 2023Massively Multilingual Shallow Fusion with Large Language Modelscited by 10
  14. 2017Sequence-to-Sequence ASR Optimization via Reinforcement Learningcited by 9
  15. 2017Reinforcement Learning of Speech Recognition System Based on Policy Gradient and Hypothesis Selectioncited by 9
  16. 2022Self-critical Sequence Training for Automatic Speech Recognitioncited by 8
  17. 2024Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Modelscited by 8
  18. 2024Applying LLMs for Rescoring N-best ASR Hypotheses of Casual Conversations: Effects of Domain Adaptation and Context Carry-overcited by 4
  19. 2021Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Maskcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond WER: Entity and Disfluency Recall in Accented Conversational ASRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning
  2. 2025Explore the Reinforcement Learning for the LLM based ASR and TTS system
  3. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  4. 2026ASSET: Adaptive Speech Segmentation via Reinforcement Learning
  5. 2023Corpus Synthesis for Zero-shot ASR domain Adaptation using Large Language Models
  6. 2026Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards
  7. 2023Multi-stage Large Language Model Correction for Speech Recognition
  8. 2025Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models