mlsanityData to 14 Oct 2026

Multi-modal Masked Siamese Network Improves Chest X-Ray Representation Learning

arXiv 2407.04449 ↗First posted 5 Jul 2024cs.CV cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning methods for medical images primarily rely on the imaging modality during pretraining. While such approaches deliver promising results, they do not leverage associated patient or scan information collected within Electronic Health Records (EHR). Here, we propose to incorporate EHR data during self-supervised pretraining with a Masked Siamese Network (MSN) to enhance the quality of chest X-ray representations. We investigate three types of EHR data, including demographic, scan metadata, and inpatient stay information. We evaluate our approach on three publicly available chest X-ray datasets, MIMIC-CXR, CheXpert, and NIH-14, using two vision transformer (ViT) backbones, specifically ViT-Tiny and ViT-Small. In assessing the quality of the representations via linear evaluation, our proposed method demonstrates significant improvement compared to vanilla MSN and state-of-the-art self-supervised learning baselines. Our work highlights the potential of EHR-enhanced self-supervised pre-training for medical imaging. The code is publicly available at: https://github.com/nyuad-cai/CXR-EHR-MSN

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 68 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

42 of 68 references.

References · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  11. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  12. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
30 more
  1. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  2. 2017Mixed Precision Trainingcited by 1,156
  3. 2016Colorful Image Colorizationcited by 1,114
  4. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  6. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  7. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  8. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  9. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  10. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  11. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  12. 2023Multimodal Deep Learningcited by 258
  13. 2022Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistencycited by 250
  14. 2020CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patientscited by 171
  15. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  16. 20203D Self-Supervised Methods for Medical Imagingcited by 134
  17. 2021Self-supervised learning methods and applications in medical imaging analysis: A surveycited by 77
  18. 2019Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cubecited by 76
  19. 2022MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray imagescited by 54
  20. 2021MedAug: Contrastive learning leveraging patient metadata improves representations for chest X-ray interpretationcited by 52
  21. 2020Momentum Contrastive Learning for Few-Shot COVID-19 Diagnosis from Chest CT Imagescited by 43
  22. 2018Multimodal Machine Learning for Automated ICD Codingcited by 41
  23. 2020PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentationcited by 36
  24. 2021COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Predictioncited by 36
  25. 2017Exploiting the potential of unlabeled endoscopic video data with self-supervised learningcited by 33
  26. 2017Self-Supervised Learning for Spinal MRIscited by 24
  27. 2021Self-Adaptive Training: Bridging Supervised and Self-Supervised Learningcited by 17
  28. 2023Exploring Image Augmentations for Siamese Representation Learning with Chest X-Rayscited by 12
  29. 2021ScatSimCLR: self-supervised contrastive learning with pretext task regularization for small-scale datasetscited by 9
  30. 2022Can we Adopt Self-supervised Pretraining for Chest X-Rays?cited by 4

Built on by

Not cited yet.