# Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech

arXiv: https://arxiv.org/abs/2407.13035 · HTML: https://mlsanity.com/paper/2407.13035/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 34 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 5 of 5)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [How Does Pre-trained Wav2Vec 2.0 Perform on Domain Shifted ASR? An Extensive Benchmark on Air Traffic Control Communications](https://mlsanity.com/paper/2203.16822/) (2022)
- [Investigating salient representations and label Variance in Dimensional Speech Emotion Analysis](https://mlsanity.com/paper/2312.16180/) (2023)
- [Investigation and Analysis of Hyper and Hypo neuron pruning to selectively update neurons during Unsupervised Adaptation](https://mlsanity.com/paper/2001.01755/) (2020)

## Built on by (most cited first, 4 of 4)

- [Modeling speech emotion with label variance and analyzing performance across speakers and unseen acoustic conditions](https://mlsanity.com/paper/2503.22711/) (2025)
- [Foundation Model Hidden Representations for Heart Rate Estimation from Auscultation](https://mlsanity.com/paper/2505.20745/) (2025)
- [Estimating Respiratory Effort from Nocturnal Breathing Sounds for Obstructive Sleep Apnoea Screening](https://mlsanity.com/paper/2509.14944/) (2025)
- [Breathing and Semantic Pause Detection and Exertion-Level Classification in Post-Exercise Speech](https://mlsanity.com/paper/2509.15473/) (2025)

## Abstract (arXiv)

The process of human speech production involves coordinated respiratory action to elicit acoustic speech signals. Typically, speech is produced when air is forced from the lungs and is modulated by the vocal tract, where such actions are interspersed by moments of breathing in air (inhalation) to refill the lungs again. Respiratory rate (RR) is a vital metric that is used to assess the overall health, fitness, and general well-being of an individual. Existing approaches to measure RR (number of breaths one takes in a minute) are performed using specialized equipment or training. Studies have demonstrated that machine learning algorithms can be used to estimate RR using bio-sensor signals as input. Speech-based estimation of RR can offer an effective approach to measure the vital metric without requiring any specialized equipment or sensors. This work investigates a machine learning based approach to estimate RR from speech segments obtained from subjects speaking to a close-talking microphone device. Data were collected from N=26 individuals, where the groundtruth RR was obtained through commercial grade chest-belts and then manually corrected for any errors. A convolutional long-short term memory network (Conv-LSTM) is proposed to estimate respiration time-series data from the speech signal. We demonstrate that the use of pre-trained representations obtained from a foundation model, such as Wav2Vec2, can be used to estimate respiration-time-series with low root-mean-squared error and high correlation coefficient, when compared with the baseline. The model-driven time series can be used to estimate RR with a low mean absolute error (MAE) ~ 1.6 breaths/min.