# Voice EHR: Introducing Multimodal Audio Data for Health

arXiv: https://arxiv.org/abs/2404.01620 · HTML: https://mlsanity.com/paper/2404.01620/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 58 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data](https://mlsanity.com/paper/2006.05919/) (2020)
- [Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infection](https://mlsanity.com/paper/2305.12741/) (2023)
- [Sounds of COVID-19: exploring realistic performance of audio-based digital testing](https://mlsanity.com/paper/2106.15523/) (2021)
- [Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers](https://mlsanity.com/paper/2212.08570/) (2022)
- [Hi Sigma, do I have the Coronavirus?: Call for a New Artificial Intelligence Approach to Support Health Care Professionals Dealing With The COVID-19 Pandemic](https://mlsanity.com/paper/2004.06510/) (2020)
- [COVID-19 Patient Detection from Telephone Quality Speech Data](https://mlsanity.com/paper/2011.04299/) (2020)
- [Analyzing the impact of SARS-CoV-2 variants on respiratory sound signals](https://mlsanity.com/paper/2206.12309/) (2022)
- [COVYT: Introducing the Coronavirus YouTube and TikTok speech dataset featuring the same speakers with and without infection](https://mlsanity.com/paper/2206.11045/) (2022)

## Built on by (most cited first, 2 of 2)

- [High-precision medical speech recognition through synthetic data and semantic correction: UNITED-MEDASR](https://mlsanity.com/paper/2412.00055/) (2024)
- [Beyond .WAV: Design and Software Verification of VocalCap, a Traceable Browser-Based Audio Capture System for Vocal Biomarker Research](https://mlsanity.com/paper/2609.03320/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) models trained on audio data may have the potential to rapidly perform clinical tasks, enhancing medical decision-making and potentially improving outcomes through early detection. Existing technologies depend on limited datasets collected with expensive recording equipment in high-income countries, which challenges deployment in resource-constrained, high-volume settings where audio data may have a profound impact on health equity. This report introduces a novel data type and a corresponding collection system that captures health data through guided questions using only a mobile/web application. The app facilitates the collection of an audio electronic health record (Voice EHR) which may contain complex biomarkers of health from conventional voice/respiratory features, speech patterns, and spoken language with semantic meaning and longitudinal context, potentially compensating for the typical limitations of unimodal clinical datasets. This report presents the application used for data collection, initial experiments on data quality, and case studies which demonstrate the potential of voice EHR to advance the scalability/diversity of audio AI.