Zero-Shot End-To-End Spoken Question Answering In Medical Domain
Not on the map. Why?
Abstract (arXiv)
In the rapidly evolving landscape of spoken question-answering (SQA), the integration of large language models (LLMs) has emerged as a transformative development. Conventional approaches often entail the use of separate models for question audio transcription and answer selection, resulting in significant resource utilization and error accumulation. To tackle these challenges, we explore the effectiveness of end-to-end (E2E) methodologies for SQA in the medical domain. Our study introduces a novel zero-shot SQA approach, compared to traditional cascade systems. Through a comprehensive evaluation conducted on a new open benchmark of 8 medical tasks and 48 hours of synthetic audio, we demonstrate that our approach requires up to 14.7 times fewer resources than a combined 1.3B parameters LLM with a 1.55B parameters ASR model while improving average accuracy by 0.5%. These findings underscore the potential of E2E methodologies for SQA in resource-constrained contexts.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 9 Jun 2024.
Builds on
28 of 33 references.
References · 28
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
- 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
16 more
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
- 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
- 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
- 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
- 2023Pengi: An Audio Language Model for Audio Taskscited by 217
- 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
- 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
- 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
- 2018Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehensioncited by 74
- 2019Multilingual End-to-End Speech Translationcited by 37
- 2020Knowledge Distillation for Improved Accuracy in Spoken Question Answeringcited by 31
- 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
- 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
- 2023Investigating Pre-trained Audio Encoders in the Low-Resource Conditioncited by 11
- 2019Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptationcited by 6
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.