# Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking

arXiv: https://arxiv.org/abs/2406.16148 · HTML: https://mlsanity.com/paper/2406.16148/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 76 references that are in the map.
- First posted on arXiv 23 Jun 2024.

## Builds on (most cited first, 30 of 39)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2304.01933/) (2023)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://mlsanity.com/paper/2402.07729/) (2024)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Prompting Large Language Models with Speech Recognition Abilities](https://mlsanity.com/paper/2307.11795/) (2023)
- [SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling](https://mlsanity.com/paper/2302.00861/) (2023)
- [BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation](https://mlsanity.com/paper/2103.06695/) (2021)
- [Large-scale Training of Foundation Models for Wearable Biosignals](https://mlsanity.com/paper/2312.05409/) (2023)
- [MAE-AST: Masked Autoencoding Audio Spectrogram Transformer](https://mlsanity.com/paper/2203.16691/) (2022)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [The COUGHVID crowdsourcing dataset: A corpus for the study of large-scale cough analysis algorithms](https://mlsanity.com/paper/2009.11644/) (2020)
- [Toward a Foundation Model for Time Series Data](https://mlsanity.com/paper/2310.03916/) (2023)
- [Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification](https://mlsanity.com/paper/2305.14032/) (2023)
- [Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Input](https://mlsanity.com/paper/2210.14648/) (2022)

## Built on by (most cited first, 23 of 23)

- [RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction](https://mlsanity.com/paper/2410.05361/) (2024)
- [ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wild](https://mlsanity.com/paper/2512.06721/) (2025)
- [CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning](https://mlsanity.com/paper/2505.01199/) (2025)
- [Shifting the Paradigm: A Diffeomorphism Between Time Series Data Manifolds for Achieving Shift-Invariancy in Deep Learning](https://mlsanity.com/paper/2502.19921/) (2025)
- [Improving Respiratory Sound Classification with Architecture-Agnostic Knowledge Distillation from Ensembles](https://mlsanity.com/paper/2505.22027/) (2025)
- [BenchHAR: Benchmarking Self-Supervised Learning for Generalizable Sensor-based Activity Recognition](https://mlsanity.com/paper/2605.08296/) (2026)
- [Towards reliable respiratory disease diagnosis based on cough sounds and vision transformers](https://mlsanity.com/paper/2408.15667/) (2024)
- [CLAP-S: Support Set Based Adaptation for Downstream Fiber-optic Acoustic Recognition](https://mlsanity.com/paper/2501.09877/) (2025)
- [Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding](https://mlsanity.com/paper/2512.04847/) (2025)
- [StethoLM: Audio Language Model for Cardiopulmonary Analysis Across Clinical Tasks](https://mlsanity.com/paper/2603.00355/) (2026)
- [Estimating Respiratory Effort from Nocturnal Breathing Sounds for Obstructive Sleep Apnoea Screening](https://mlsanity.com/paper/2509.14944/) (2025)
- [DomusFM: A Foundation Model for Event-Based Behavioral Monitoring in Smart-Homes](https://mlsanity.com/paper/2602.01910/) (2026)
- [Automated Dysphagia Screening Using Noninvasive Neck Acoustic Sensing](https://mlsanity.com/paper/2602.02725/) (2026)
- [Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis](https://mlsanity.com/paper/2602.15909/) (2026)
- [RA-QA: A Benchmarking System for Respiratory Audio Question Answering Under Real-World Heterogeneity](https://mlsanity.com/paper/2602.18452/) (2026)
- [RAMoEA-QA: Hierarchical Specialization for Robust Respiratory Audio Question Answering](https://mlsanity.com/paper/2603.06542/) (2026)
- [Patient-Level Multimodal Question Answering from Multi-Site Auscultation Recordings](https://mlsanity.com/paper/2603.13362/) (2026)
- [Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classification](https://mlsanity.com/paper/2604.24096/) (2026)
- [CoughSense: Five-Class Respiratory Disease Classification via Whisper Encoder Fine-Tuning and Dual-Encoder Cross-Attention Fusion with Balanced Contrastive Learning](https://mlsanity.com/paper/2606.02998/) (2026)
- [Beyond Classification: A Cough Regression Benchmark for Respiratory Acoustic Foundation Models](https://mlsanity.com/paper/2606.15436/) (2026)
- [BCoughBench: Benchmarking Respiratory Acoustic Foundation Models Under Body-Coupled Wearable Sensor Conditions](https://mlsanity.com/paper/2606.25116/) (2026)
- [Why ML-based cough models do not generalize: a systematic cross-dataset evaluation for tuberculosis screening](https://mlsanity.com/paper/2608.25846/) (2026)
- [Zero-Shot Respiratory Sound Classification through LLM-Augmented Audio-Text Alignment](https://mlsanity.com/paper/2609.00055/) (2026)

## Abstract (arXiv)

Respiratory audio, such as coughing and breathing sounds, has predictive power for a wide range of healthcare applications, yet is currently under-explored. The main problem for those applications arises from the difficulty in collecting large labeled task-specific data for model development. Generalizable respiratory acoustic foundation models pretrained with unlabeled data would offer appealing advantages and possibly unlock this impasse. However, given the safety-critical nature of healthcare applications, it is pivotal to also ensure openness and replicability for any proposed foundation model solution. To this end, we introduce OPERA, an OPEn Respiratory Acoustic foundation model pretraining and benchmarking system, as the first approach answering this need. We curate large-scale respiratory audio datasets (~136K samples, over 400 hours), pretrain three pioneering foundation models, and build a benchmark consisting of 19 downstream respiratory health tasks for evaluation. Our pretrained models demonstrate superior performance (against existing acoustic models pretrained with general audio on 16 out of 19 tasks) and generalizability (to unseen datasets and new respiratory audio modalities). This highlights the great promise of respiratory acoustic foundation models and encourages more studies using OPERA as an open resource to accelerate research on respiratory audio for health. The system is accessible from https://github.com/evelyn0414/OPERA.