# Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation

arXiv: https://arxiv.org/abs/2505.20745 · HTML: https://mlsanity.com/paper/2505.20745/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 29 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 15 of 15)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Speech Emotion Recognition using Self-Supervised Features](https://mlsanity.com/paper/2202.03896/) (2022)
- [HeAR -- Health Acoustic Representations](https://mlsanity.com/paper/2403.02522/) (2024)
- [Short-segment heart sound classification using an ensemble of deep convolutional neural networks](https://mlsanity.com/paper/1810.11573/) (2018)
- [Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis](https://mlsanity.com/paper/2312.16180/) (2023)
- [Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis](https://mlsanity.com/paper/2303.03177/) (2023)
- [Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech](https://mlsanity.com/paper/2407.13035/) (2024)
- [An Ensemble of Transfer, Semi-supervised and Supervised Learning Methods for Pathological Heart Sound Classification](https://mlsanity.com/paper/1806.06506/) (2018)
- [Predicting Heart Activity from Speech using Data-driven and Knowledge-based features](https://mlsanity.com/paper/2406.06341/) (2024)
- [Model-driven Heart Rate Estimation and Heart Murmur Detection based on Phonocardiogram](https://mlsanity.com/paper/2407.18424/) (2024)

## Built on by (most cited first, 1 of 1)

- [Breathing and Semantic Pause Detection and Exertion-Level Classification in Post-Exercise Speech](https://mlsanity.com/paper/2509.15473/) (2025)

## Abstract (arXiv)

Auscultation, particularly heart sound, is a non-invasive technique that provides essential vital sign information. Recently, self-supervised acoustic representation foundation models (FMs) have been proposed to offer insights into acoustics-based vital signs. However, there has been little exploration of the extent to which auscultation is encoded in these pre-trained FM representations. In this work, using a publicly available phonocardiogram (PCG) dataset and a heart rate (HR) estimation model, we conduct a layer-wise investigation of six acoustic representation FMs: HuBERT, wav2vec2, wavLM, Whisper, Contrastive Language-Audio Pretraining (CLAP), and an in-house CLAP model. Additionally, we implement the baseline method from Nie et al., 2024 (which relies on acoustic features) and show that overall, representation vectors from pre-trained foundation models (FMs) offer comparable performance to the baseline. Notably, HR estimation using the representations from the audio encoder of the in-house CLAP model outperforms the results obtained from the baseline, achieving a lower mean absolute error (MAE) across various train/validation/test splits despite the domain mismatch.