# Text-only adaptation in LLM-based ASR through text denoising

arXiv: https://arxiv.org/abs/2601.20900 · HTML: https://mlsanity.com/paper/2601.20900/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 27 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Connecting Speech Encoder and Large Language Model for ASR](https://mlsanity.com/paper/2309.13963/) (2023)
- [Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting](https://mlsanity.com/paper/2309.15649/) (2023)
- [SLM: Bridge the thin gap between speech and text foundation models](https://mlsanity.com/paper/2310.00230/) (2023)
- [SpeechVerse: A Large-scale Generalizable Audio Language Model](https://mlsanity.com/paper/2405.08295/) (2024)
- [Can Generative Large Language Models Perform ASR Error Correction?](https://mlsanity.com/paper/2307.04172/) (2023)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognition](https://mlsanity.com/paper/2306.16007/) (2023)
- [SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus](https://mlsanity.com/paper/2309.05396/) (2023)
- [Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward](https://mlsanity.com/paper/2411.03866/) (2024)
- [Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets](https://mlsanity.com/paper/2405.02132/) (2024)
- [Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study](https://mlsanity.com/paper/2401.12789/) (2024)
- [Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning](https://mlsanity.com/paper/2506.05671/) (2025)
- [Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning](https://mlsanity.com/paper/2412.06967/) (2024)
- [Bridging the Modality Gap: Softly Discretizing Audio Representation for LLM-based Automatic Speech Recognition](https://mlsanity.com/paper/2506.05706/) (2025)
- [Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs](https://mlsanity.com/paper/2506.08633/) (2025)

## Built on by (most cited first, 8 of 8)

- [Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR](https://mlsanity.com/paper/2603.26246/) (2026)
- [On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASR](https://mlsanity.com/paper/2603.27981/) (2026)
- [How to Leverage Synthetic Speech for LLM-Based ASR Systems?](https://mlsanity.com/paper/2606.29031/) (2026)
- [Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR](https://mlsanity.com/paper/2604.06487/) (2026)
- [TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMs](https://mlsanity.com/paper/2604.08384/) (2026)
- [CORTIS: Text-Only Adaptation of Spoken Language Models for Task-Oriented Voice Agents](https://mlsanity.com/paper/2606.21453/) (2026)
- [When Synthetic Speech Is All You Have: Better Call GRPO](https://mlsanity.com/paper/2607.08409/) (2026)
- [Voice Memory for Agentic Speech Recognition](https://mlsanity.com/paper/2607.26410/) (2026)

## Abstract (arXiv)

Adapting large language model (LLM)-based automatic speech recognition (ASR) systems to new domains using text-only data is a significant yet underexplored challenge. Standard fine-tuning of the LLM on the target domain text often disrupts the critical alignment between the speech and text modality learned by the projector, degrading performance. We introduce a novel text-only adaptation method that frames this process as a text denoising task. Our approach trains the LLM to recover clean transcripts from noisy inputs. This process effectively adapts the model to a target domain while preserving cross-modal alignment. Our solution is lightweight, requiring no architectural changes or additional parameters. Extensive evaluation on two datasets demonstrates up to 22.1% relative improvement, outperforming recent state-of-the-art text-only adaptation methods.