# Building an ASR Error Robust Spoken Virtual Patient System in a Highly Class-Imbalanced Scenario Without Speech Data

arXiv: https://arxiv.org/abs/2204.05183 · HTML: https://mlsanity.com/paper/2204.05183/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [ASR Error Correction and Domain Adaptation Using Machine Translation](https://mlsanity.com/paper/2003.07692/) (2020)
- [Hallucination of speech recognition errors with sequence to sequence learning](https://mlsanity.com/paper/2103.12258/) (2021)
- [Warped Language Models for Noise Robust Language Understanding](https://mlsanity.com/paper/2011.01900/) (2020)
- [Handling Class Imbalance in Low-Resource Dialogue Systems by Combining Few-Shot Classification and Interpolation](https://mlsanity.com/paper/2010.15090/) (2020)

## Built on by (most cited first, 1 of 1)

- [Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models](https://mlsanity.com/paper/2502.12414/) (2025)

## Abstract (arXiv)

A Virtual Patient (VP) is a powerful tool for training medical students to take patient histories, where responding to a diverse set of spoken questions is essential to simulate natural conversations with a student. The performance of such a Spoken Language Understanding system (SLU) can be adversely affected by both the presence of Automatic Speech Recognition (ASR) errors in the test data and a high degree of class imbalance in the SLU training data. While these two issues have been addressed separately in prior work, we develop a novel two-step training methodology that tackles both these issues effectively in a single dialog agent. As it is difficult to collect spoken data from users without a functioning SLU system, our method does not rely on spoken data for training, rather we use an ASR error predictor to "speechify" the text data. Our method shows significant improvements over strong baselines on the VP intent classification task at various word error rate settings.