# HeAR -- Health Acoustic Representations

arXiv: https://arxiv.org/abs/2403.02522 · HTML: https://mlsanity.com/paper/2403.02522/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 57 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 38)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [Self-supervised Learning with Random-projection Quantizer for Speech Recognition](https://mlsanity.com/paper/2202.01855/) (2022)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [The COUGHVID crowdsourcing dataset: A corpus for the study of large-scale cough analysis algorithms](https://mlsanity.com/paper/2009.11644/) (2020)
- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders](https://mlsanity.com/paper/2308.01317/) (2023)
- [Multimodal LLMs for health grounded in individual-specific data](https://mlsanity.com/paper/2307.09018/) (2023)
- [Trainable Frontend For Robust and Far-Field Keyword Spotting](https://mlsanity.com/paper/1607.05666/) (2016)

## Built on by (most cited first, 15 of 15)

- [Scaling Wearable Foundation Models](https://mlsanity.com/paper/2410.13638/) (2024)
- [Towards Open Respiratory Acoustic Foundation Models: Pretraining and Benchmarking](https://mlsanity.com/paper/2406.16148/) (2024)
- [Health AI Developer Foundations](https://mlsanity.com/paper/2411.15128/) (2024)
- [Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation](https://mlsanity.com/paper/2505.20745/) (2025)
- [XAI-Driven Spectral Analysis of Cough Sounds for Respiratory Disease Characterization](https://mlsanity.com/paper/2508.14949/) (2025)
- [Pediatric Asthma Detection with Googles HeAR Model: An AI-Driven Respiratory Sound Classifier](https://mlsanity.com/paper/2504.20124/) (2025)
- [Automatic Cough Analysis for Non-Small Cell Lung Cancer Detection](https://mlsanity.com/paper/2507.19174/) (2025)
- [Estimating Respiratory Effort from Nocturnal Breathing Sounds for Obstructive Sleep Apnoea Screening](https://mlsanity.com/paper/2509.14944/) (2025)
- [AI-Driven Acoustic Voice Biomarker-Based Hierarchical Classification of Benign Laryngeal Voice Disorders from Sustained Vowels](https://mlsanity.com/paper/2512.24628/) (2025)
- [Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis](https://mlsanity.com/paper/2602.15909/) (2026)
- [Patient-level validation of a foundation-model pipeline for pediatric lung sounds: physician-annotated adventitious events are recognized, disease groups are not reliably predicted](https://mlsanity.com/paper/2603.15688/) (2026)
- [Beyond Classification: A Cough Regression Benchmark for Respiratory Acoustic Foundation Models](https://mlsanity.com/paper/2606.15436/) (2026)
- [Unlocking In-Context Learning in Audio-Language Models from Decentralized Medical Audio](https://mlsanity.com/paper/2606.23243/) (2026)
- [BCoughBench: Benchmarking Respiratory Acoustic Foundation Models Under Body-Coupled Wearable Sensor Conditions](https://mlsanity.com/paper/2606.25116/) (2026)
- [Why ML-based cough models do not generalize: a systematic cross-dataset evaluation for tuberculosis screening](https://mlsanity.com/paper/2608.25846/) (2026)

## Abstract (arXiv)

Health acoustic sounds such as coughs and breaths are known to contain useful health signals with significant potential for monitoring health and disease, yet are underexplored in the medical machine learning community. The existing deep learning systems for health acoustics are often narrowly trained and evaluated on a single task, which is limited by data and may hinder generalization to other tasks. To mitigate these gaps, we develop HeAR, a scalable self-supervised learning-based deep learning system using masked autoencoders trained on a large dataset of 313 million two-second long audio clips. Through linear probes, we establish HeAR as a state-of-the-art health audio embedding model on a benchmark of 33 health acoustic tasks across 6 datasets. By introducing this work, we hope to enable and accelerate further health acoustics research.