# RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

arXiv: https://arxiv.org/abs/2410.05361 · HTML: https://mlsanity.com/paper/2410.05361/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Can large language models reason about medical questions?](https://mlsanity.com/paper/2207.08143/) (2022)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [LLMs Accelerate Annotation for Medical Information Extraction](https://mlsanity.com/paper/2312.02296/) (2023)
- [Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement](https://mlsanity.com/paper/2403.06659/) (2024)
- [MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images](https://mlsanity.com/paper/2207.07027/) (2022)
- [Instruction-tuning Aligns LLMs to the Human Brain](https://mlsanity.com/paper/2312.00575/) (2023)
- [RespireNet: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting](https://mlsanity.com/paper/2011.00196/) (2020)
- [Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking](https://mlsanity.com/paper/2406.16148/) (2024)
- [Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infection](https://mlsanity.com/paper/2305.12741/) (2023)
- [Exploring Automatic COVID-19 Diagnosis via voice and symptoms from Crowdsourced Data](https://mlsanity.com/paper/2102.05225/) (2021)
- [Pretraining Respiratory Sound Representations using Metadata and Contrastive Learning](https://mlsanity.com/paper/2210.16192/) (2022)
- [Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers](https://mlsanity.com/paper/2212.08570/) (2022)
- [BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification](https://mlsanity.com/paper/2406.06786/) (2024)
- [Fusion Strategies for Learning User Embeddings with Neural Networks](https://mlsanity.com/paper/1901.02322/) (2019)
- [Cross-device Federated Learning for Mobile Health Diagnostics: A First Study on COVID-19 Detection](https://mlsanity.com/paper/2303.07067/) (2023)

## Built on by (most cited first, 10 of 10)

- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [Wearable Foundation Models Should Go Beyond Static Encoders](https://mlsanity.com/paper/2603.19564/) (2026)
- [Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding](https://mlsanity.com/paper/2512.04847/) (2025)
- [StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks](https://mlsanity.com/paper/2603.00355/) (2026)
- [Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis](https://mlsanity.com/paper/2602.15909/) (2026)
- [RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity](https://mlsanity.com/paper/2602.18452/) (2026)
- [Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings](https://mlsanity.com/paper/2603.13362/) (2026)
- [Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classification](https://mlsanity.com/paper/2604.12647/) (2026)
- [Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment](https://mlsanity.com/paper/2609.00055/) (2026)

## Abstract (arXiv)

The high incidence and mortality rates associated with respiratory diseases underscores the importance of early screening. Machine learning models can automate clinical consultations and auscultation, offering vital support in this area. However, the data involved, spanning demographics, medical history, symptoms, and respiratory audio, are heterogeneous and complex. Existing approaches are insufficient and lack generalizability, as they typically rely on limited training data, basic fusion techniques, and task-specific models. In this paper, we propose RespLLM, a novel multimodal large language model (LLM) framework that unifies text and audio representations for respiratory health prediction. RespLLM leverages the extensive prior knowledge of pretrained LLMs and enables effective audio-text fusion through cross-modal attentions. Instruction tuning is employed to integrate diverse data from multiple sources, ensuring generalizability and versatility of the model. Experiments on five real-world datasets demonstrate that RespLLM outperforms leading baselines by an average of 4.6% on trained tasks, 7.9% on unseen datasets, and facilitates zero-shot predictions for new tasks. Our work lays the foundation for multimodal models that can perceive, listen to, and understand heterogeneous data, paving the way for scalable respiratory health diagnosis.