mlsanityData to 14 Oct 2026

BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis

arXiv 2108.04938 ↗First posted 10 Aug 2021cs.CV cs.AI cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Vision-and-language(V&L) models take image and text as input and learn to capture the associations between them. Prior studies show that pre-trained V&L models can significantly improve the model performance for downstream tasks such as Visual Question Answering (VQA). However, V&L models are less effective when applied in the medical domain (e.g., on X-ray images and clinical notes) due to the domain gap. In this paper, we investigate the challenges of applying pre-trained V&L models in medical applications. In particular, we identify that the visual representation in general V&L models is not suitable for processing medical data. To overcome this limitation, we propose BERTHop, a transformer-based model based on PixelHop++ and VisualBERT, for better capturing the associations between the two modalities. Experiments on the OpenI dataset, a commonly used thoracic disease diagnosis benchmark, show that BERTHop achieves an average Area Under the Curve (AUC) of 98.12% which is 1.62% higher than state-of-the-art (SOTA) while it is trained on a 9 times smaller dataset.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 10 Aug 2021.

Builds on

33 of 45 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
21 more
  1. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  2. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  3. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  4. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  5. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 461
  6. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 309
  7. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  8. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  9. 2016Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learningcited by 209
  10. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  11. 2017NegBio: a high-performance tool for negation and uncertainty detection in radiology reportscited by 79
  12. 2018Interpretable Convolutional Neural Networks via Feedforward Designcited by 49
  13. 2020PixelHop++: A Small Successive-Subspace-Learning-Based (SSL-based) Model for Image Classificationcited by 37
  14. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  15. 2021DefakeHop: A Light-Weight High-Performance Deepfake Detectorcited by 31
  16. 2021VisualCheXbert: Addressing the Discrepancy Between Radiology Report Labels and Image Labelscited by 27
  17. 2021VoxelHop: Successive Subspace Learning for ALS Disease Classification Using Structural MRIcited by 18
  18. 2020FaceHop: A Light-Weight Low-Resolution Face Gender Classification Methodcited by 16
  19. 2020Visual Question Answering on 360deg Imagescited by 15
  20. 2021Successive Subspace Learning: An Overviewcited by 11
  21. 2020Low-Resolution Face Recognition In Resource-Constrained Environmentscited by 10

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 5; 2024: 5; 2025: 5.

20 papers.

Sort:

By area:

  1. 2023Large language models in healthcare and medical domain: A reviewcited by 127
  2. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  3. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  4. 2022Transformers in Medical Image Analysis: A Reviewcited by 41
  5. 2023Advancing Medical Imaging with Language Models: A Journey from N-grams to ChatGPTcited by 28
  6. 2024MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosiscited by 28
  7. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  8. 2024Tuning-Free Accountable Intervention for LLM Deployment -- A Metacognitive Approachcited by 11
  9. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  10. 2023Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentationcited by 10
  11. 2024Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Groundingcited by 10
  12. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
8 more
  1. 2023Several categories of Large Language Models (LLMs): A Short Surveycited by 7
  2. 2022DefakeHop++: An Enhanced Lightweight Deepfake Detectorcited by 5
  3. 2022PAGER: Progressive Attribute-Guided Extendable Robust Image Generationcited by 5
  4. 2025Recent Advances in Medical Image Classificationcited by 2
  5. 2022The Ability of Image-Language Explainable Models to Resemble Domain Expertisecited by 1
  6. 2025ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rayscited by 1
  7. 2025An Explainable Transformer Model for Alzheimer's Disease Detection Using Retinal Imagingcited by 1
  8. 2025Revisiting Performance Claims for Chest X-Ray Models Using Clinical Contextcited by 1