mlsanityData to 14 Oct 2026

MMBERT: Multimodal BERT Pretraining for Improved Medical VQA

arXiv 2104.01394 ↗First posted 3 Apr 2021cs.CV cs.CL cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Images in the medical domain are fundamentally different from the general domain images. Consequently, it is infeasible to directly employ general domain Visual Question Answering (VQA) models for the medical domain. Additionally, medical images annotation is a costly and time-consuming process. To overcome these limitations, we propose a solution inspired by self-supervised pretraining of Transformer-style architectures for NLP, Vision and Language tasks. Our method involves learning richer medical image and text semantic representations using Masked Language Modeling (MLM) with image features as the pretext task on a large medical image+caption dataset. The proposed solution achieves new state-of-the-art performance on two VQA datasets for radiology images -- VQA-Med 2019 and VQA-RAD, outperforming even the ensemble models of previous best solutions. Moreover, our solution provides attention maps which help in model interpretability. The code is available at https://github.com/VirajBagal/MMBERT

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 7 of its 14 references that are in the map. First posted on arXiv 3 Apr 2021.

Builds on

7 of 14 references.

References · 7

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  7. 2019Overcoming Data Limitation in Medical Visual Question Answeringcited by 71

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 26; 2024: 13; 2025: 18; 2026: 3.

68 papers.

Sort:

By area:

  1. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  2. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  3. 2021Medical Visual Question Answering: A Surveycited by 93
  4. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  5. 2025MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflowcited by 73
  6. 2023Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Modelscited by 58
  7. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 57
  8. 2022Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledgecited by 51
  9. 2024MedCoT: Medical Chain of Thought via Hierarchical Expertcited by 46
  10. 2023ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Modelcited by 40
  11. 2023CLIP in Medical Imaging: A Surveycited by 40
  12. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
38 more
  1. 2023Towards Unifying Medical Vision-and-Language Pre-training via Soft Promptscited by 36
  2. 2023RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-trainingcited by 31
  3. 2023Q2ATransformer: Improving Medical VQA via an Answer Querying Decodercited by 31
  4. 2022Self-supervised vision-language pretraining for Medical visual question answeringcited by 30
  5. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 29
  6. 2023Medical Vision Language Pretraining: A surveycited by 24
  7. 2024Kvasir-VQA: A Text-Image Pair GI Tract Datasetcited by 23
  8. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-trainingcited by 22
  9. 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
  10. 2023Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reportingcited by 18
  11. 2022DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queriescited by 17
  12. 2023Retrieval-based Knowledge Augmented Vision Language Pre-trainingcited by 17
  13. 2024Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgerycited by 17
  14. 2025SARLANG-1M: A Benchmark for Vision-Language Modeling in SAR Image Understandingcited by 16
  15. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  16. 2024LaPA: Latent Prompt Assist Model For Medical Visual Question Answeringcited by 12
  17. 2023Hierarchical Pretraining on Multimodal Electronic Health Recordscited by 11
  18. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  19. 2023LiT-4-RSVQA: Lightweight Transformer-based Visual Question Answering in Remote Sensingcited by 10
  20. 2023Visual Question Answering in the Medical Domaincited by 8
  21. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practicecited by 8
  22. 2022UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answeringcited by 6
  23. 2023One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Datacited by 6
  24. 2022A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answeringcited by 5
  25. 2023Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmarkcited by 5
  26. 2023MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generationcited by 5
  27. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  28. 2025SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imagingcited by 5
  29. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
  30. 2023Domain-adapted large language models for classifying nuclear medicine reportscited by 4
  31. 2023Multimodal Contrastive Learning with Hard Negative Sampling for Human Activity Recognitioncited by 4
  32. 2023A Survey on Image-text Multimodal Modelscited by 4
  33. 2023BESTMVQA: A Benchmark Evaluation System for Medical Visual Question Answeringcited by 3
  34. 2023UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Promptscited by 3
  35. 2024Unified Multi-modal Diagnostic Framework with Reconstruction Pre-training and Heterogeneity-combat Tuningcited by 3
  36. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answeringcited by 3
  37. 2025Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reportscited by 3
  38. 2025Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQAcited by 3

50 most cited of 68 All 68 (JSON)