# BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound Classification

arXiv: https://arxiv.org/abs/2406.06786 · HTML: https://mlsanity.com/paper/2406.06786/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification](https://mlsanity.com/paper/2112.02690/) (2021)
- [RespireNet: A Deep Neural Network for Accurately Detecting Abnormal Lung Sounds in Limited Data Setting](https://mlsanity.com/paper/2011.00196/) (2020)
- [Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification](https://mlsanity.com/paper/2305.14032/) (2023)
- [Prototype Learning for Interpretable Respiratory Sound Analysis](https://mlsanity.com/paper/2110.03536/) (2021)
- [Stethoscope-guided Supervised Contrastive Learning for Cross-domain Adaptation on Respiratory Sound Classification](https://mlsanity.com/paper/2312.09603/) (2023)
- [Pretraining Respiratory Sound Representations using Metadata and Contrastive Learning](https://mlsanity.com/paper/2210.16192/) (2022)
- [Lung Sound Classification Using Co-tuning and Stochastic Normalization](https://mlsanity.com/paper/2108.01991/) (2021)
- [Decoding EEG Brain Activity for Multi-Modal Natural Language Processing](https://mlsanity.com/paper/2102.08655/) (2021)
- [Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis](https://mlsanity.com/paper/2203.16141/) (2022)
- [Can Brain Signals Reveal Inner Alignment with Human Languages?](https://mlsanity.com/paper/2208.06348/) (2022)
- [RepAugment: Input-Agnostic Representation-Level Augmentation for Respiratory Sound Classification](https://mlsanity.com/paper/2405.02996/) (2024)
- [Adversarial Fine-tuning using Generated Respiratory Sound to Address Class Imbalance](https://mlsanity.com/paper/2311.06480/) (2023)

## Built on by (most cited first, 12 of 12)

- [RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction](https://mlsanity.com/paper/2410.05361/) (2024)
- [Tri-MTL: A Triple Multitask Learning Approach for Respiratory Disease Diagnosis](https://mlsanity.com/paper/2505.06271/) (2025)
- [Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles](https://mlsanity.com/paper/2505.22027/) (2025)
- [Patient-Aware Feature Alignment for Robust Lung Sound Classification:Cohesion-Separation and Global Alignment Losses](https://mlsanity.com/paper/2505.23834/) (2025)
- [Lungmix: A Mixup-Based Strategy for Generalization in Respiratory Sound Classification](https://mlsanity.com/paper/2501.00064/) (2024)
- [Mitigating Stethoscope-Induced Shortcuts in Respiratory Sound Classification under Federated Domain Generalization with Causality-Inspired Interventions](https://mlsanity.com/paper/2605.29862/) (2026)
- [Improving Deep Learning-based Respiratory Sound Analysis with Frequency Selection and Attention Mechanism](https://mlsanity.com/paper/2507.20052/) (2025)
- [Geometry-Aware Optimization for Respiratory Sound Classification: Enhancing Sensitivity with SAM-Optimized Audio Spectrogram Transformers](https://mlsanity.com/paper/2512.22564/) (2025)
- [Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis](https://mlsanity.com/paper/2602.15909/) (2026)
- [Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classification](https://mlsanity.com/paper/2604.24096/) (2026)
- [Quality Adaptive Angular Margin Learning for Respiratory Sound Classification](https://mlsanity.com/paper/2606.11915/) (2026)
- [Lung-SRAD: Spectral-Aware Regularized Audio DASS with Dual-Axis Patch-Mix Contrastive Learning for Respiratory Sound Classification](https://mlsanity.com/paper/2606.11922/) (2026)

## Abstract (arXiv)

Respiratory sound classification (RSC) is challenging due to varied acoustic signatures, primarily influenced by patient demographics and recording environments. To address this issue, we introduce a text-audio multimodal model that utilizes metadata of respiratory sounds, which provides useful complementary information for RSC. Specifically, we fine-tune a pretrained text-audio multimodal model using free-text descriptions derived from the sound samples' metadata which includes the gender and age of patients, type of recording devices, and recording location on the patient's body. Our method achieves state-of-the-art performance on the ICBHI dataset, surpassing the previous best result by a notable margin of 1.17%. This result validates the effectiveness of leveraging metadata and respiratory sound samples in enhancing RSC performance. Additionally, we investigate the model performance in the case where metadata is partially unavailable, which may occur in real-world clinical setting.