mlsanityData to 7 Oct 2026

Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning

arXiv 2501.00039 ↗First posted 25 Dec 2024eess.AS cs.CL cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We introduce a large language model (LLM) capable of processing speech inputs and show that tuning it further with reinforcement learning on human preference (RLHF) enables it to adapt better to disordered speech than traditional fine-tuning. Our method replaces low-frequency text tokens in an LLM's vocabulary with audio tokens and enables the model to recognize speech by fine-tuning it on speech with transcripts. We then use RL with rewards based on syntactic and semantic accuracy measures generalizing the LLM further to recognize disordered speech. While the resulting LLM does not outperform existing systems for speech recognition, we find that tuning with reinforcement learning using custom rewards leads to substantially better performance than supervised fine-tuning of the language model, specifically when adapting to speech in a different setting. This presents a compelling alternative tuning strategy for speech recognition using large language models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 35 references that are in the map. First posted on arXiv 25 Dec 2024.

Builds on

27 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2024Mixtral of Expertscited by 1,785
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  12. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
14 more
  1. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  2. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  4. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  5. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  6. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  7. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 40
  8. 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
  9. 2022Reward Gaming in Conditional Text Generationcited by 29
  10. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  11. 2021Adapting GPT, GPT-2 and BERT Language Models for Speech Recognitioncited by 16
  12. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14
  13. 2022Assessing ASR Model Quality on Disordered Speech using BERTScorecited by 8
  14. 2023Clinical BERTScore: An Improved Measure of Automatic Speech Recognition Performance in Clinical Settingscited by 8

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversationcited by 25
  2. 2025The Sound of Syntax: Finetuning and Comprehensive Evaluation of Language Models for Speech Pathologycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Customizing Speech Recognition Model with Large Language Model Feedback
  2. 2025Explore the Reinforcement Learning for the LLM based ASR and TTS system
  3. 2026LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference
  4. 2024Reinforcement Learning Enhanced LLMs: A Survey
  5. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  6. 2024Reinforcement Learning for LLM Post-Training: A Survey
  7. 2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
  8. 2026Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs