# SpeechMedAssist: Efficiently and Effectively Adapting Speech Language Models for Medical Consultation

arXiv: https://arxiv.org/abs/2601.04638 · HTML: https://mlsanity.com/paper/2601.04638/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2026.

## Built on by (most cited first, 2 of 2)

- [APTER: Adaptive Post-Training with Expert-Grounded Rubrics](https://mlsanity.com/paper/2608.14212/) (2026)
- [Scalable Context Orchestration for Serving LLMs Over Voice](https://mlsanity.com/paper/2609.04288/) (2026)

## Abstract (arXiv)

Medical consultations are intrinsically speech-centric. However, most prior works focus on long-text-based interactions, which are cumbersome and patient-unfriendly. Recent advances in speech language models (SpeechLMs) have enabled more natural speech-based interaction, yet the scarcity of medical speech data and the inefficiency of directly fine-tuning on speech data jointly hinder the adoption of SpeechLMs in medical consultation. In this paper, we propose SpeechMedAssist, a SpeechLM natively capable of conducting speech-based multi-turn interactions with patients. By exploiting the architectural properties of SpeechLMs, we decouple the conventional one-stage training into a two-stage paradigm consisting of (1) Knowledge & Capability Injection via Text and (2) Modality Re-alignment with Limited Speech Data, thereby reducing the requirement for medical speech data to only 10k synthesized samples. To evaluate SpeechLMs for medical consultation scenarios, we design a benchmark comprising both single-turn question answering and multi-turn simulated interactions. Experimental results show that our model outperforms all baselines in both effectiveness and robustness in most evaluation settings.