# Predicting Heart Activity from Speech using Data-driven and Knowledge-based features

arXiv: https://arxiv.org/abs/2406.06341 · HTML: https://mlsanity.com/paper/2406.06341/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation](https://mlsanity.com/paper/2103.06695/) (2021)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress](https://mlsanity.com/paper/2104.07123/) (2021)
- [The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress](https://mlsanity.com/paper/2207.05691/) (2022)
- [SERAB: A multi-lingual benchmark for speech emotion recognition](https://mlsanity.com/paper/2110.03414/) (2021)
- [BYOL-S: Learning Self-supervised Speech Representations by Bootstrapping](https://mlsanity.com/paper/2206.12038/) (2022)
- [Hybrid Handcrafted and Learnable Audio Representation for Analysis of Speech Under Cognitive and Physical Load](https://mlsanity.com/paper/2203.16637/) (2022)
- [Speech Signal Analysis for the Estimation of Heart Rates Under Different Emotional States](https://mlsanity.com/paper/1608.03720/) (2016)

## Built on by (most cited first, 2 of 2)

- [Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation](https://mlsanity.com/paper/2505.20745/) (2025)
- [Towards Robust Assessment of Pathological Voices via Combined Low-Level Descriptors and Foundation Model Representations](https://mlsanity.com/paper/2505.21356/) (2025)

## Abstract (arXiv)

Accurately predicting heart activity and other biological signals is crucial for diagnosis and monitoring. Given that speech is an outcome of multiple physiological systems, a significant body of work studied the acoustic correlates of heart activity. Recently, self-supervised models have excelled in speech-related tasks compared to traditional acoustic methods. However, the robustness of data-driven representations in predicting heart activity remained unexplored. In this study, we demonstrate that self-supervised speech models outperform acoustic features in predicting heart activity parameters. We also emphasize the impact of individual variability on model generalizability. These findings underscore the value of data-driven representations in such tasks and the need for more speech-based physiological data to mitigate speaker-related challenges.