mlsanityData to 7 Oct 2026

Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model

arXiv 2303.00091 ↗First posted 27 Feb 2023eess.AS cs.AI cs.CLcited by 6

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 54 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

27 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training24 May

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  12. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
14 more
  1. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  2. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  3. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  4. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  5. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  6. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 96
  7. 2017NegBio: a high-performance tool for negation and uncertainty detection in radiology reportscited by 79
  8. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 75
  9. 2016Wav2Letter: an End-to-End ConvNet-based Speech Recognition Systemcited by 43
  10. 2018wav2letter++: The Fastest Open-source Speech Recognition Systemcited by 42
  11. 2018End-to-end Speech Recognition with Word-based RNN Language Modelscited by 23
  12. 2018Fully Convolutional Speech Recognitioncited by 19
  13. 2013Improvements to deep convolutional neural networks for LVCSRcited by 11
  14. 2020Weak-Attention Suppression For Transformer Based Speech Recognitioncited by 7

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 4.

6 papers.

  1. 2023ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Modelcited by 40
  2. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representationscited by 11
  3. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  4. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  5. 2025Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Explorationcited by 1
  6. 2024Whispy: Adapting STT Whisper Models to Real-Time Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark
  2. 2025SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imaging
  3. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models
  4. 2023Medical Vision Language Pretraining: A survey
  5. 2023MedBLIP: Bootstrapping Language-Image Pre-training from 3D Medical Images and Texts
  6. 2023Text Generation with Speech Synthesis for ASR Data Augmentation
  7. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?
  8. 2024Medical Vision-Language Pre-Training for Brain Abnormalities