mlsanityData to 14 Oct 2026

Advancing Radiograph Representation Learning with Masked Record Modeling

arXiv 2301.13155 ↗First posted 30 Jan 2023cs.CV cs.CL cs.LGcited by 75

Not on the map. Why?

Cited by 34 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern studies in radiograph representation learning rely on either self-supervision to encode invariant semantics or associated radiology reports to incorporate medical expertise, while the complementarity between them is barely noticed. To explore this, we formulate the self- and report-completion as two complementary objectives and present a unified framework based on masked record modeling (MRM). In practice, MRM reconstructs masked image patches and masked report tokens following a multi-task scheme to learn knowledge-enhanced semantic representations. With MRM pre-training, we obtain pre-trained models that can be well transferred to various radiography tasks. Specifically, we find that MRM offers superior performance in label-efficient fine-tuning. For instance, MRM achieves 88.5% mean AUC on CheXpert using 1% labeled data, outperforming previous R2L methods with 100% labels. On NIH ChestX-ray, MRM outperforms the best performing counterpart by about 3% under small labeling ratios. Besides, MRM surpasses self- and report-supervised pre-training in identifying the pneumonia type and the pneumothorax area, sometimes by large margins.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

37 of 45 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  12. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
25 more
  1. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 641
  2. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  3. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  4. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  5. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  6. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  7. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  8. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 309
  9. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  10. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  11. 2020Learning Visual Representations with Caption Annotationscited by 123
  12. 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 109
  13. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  14. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  15. 2015Learning to See by Movingcited by 102
  16. 2020COVID-19 Image Data Collection: Prospective Predictions Are the Futurecited by 100
  17. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
  18. 2020Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representationscited by 69
  19. 2021Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learningcited by 68
  20. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 67
  21. 2021Preservational Learning Improves Self-supervised Medical Image Models by Reconstructing Diverse Contextscited by 60
  22. 2023PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysiscited by 40
  23. 2021Multimodal Representation Learning via Maximization of Local Mutual Informationcited by 27
  24. 2022Diagnose Like a Radiologist: Hybrid Neuro-Probabilistic Reasoning for Attribute-Based Medical Image Diagnosiscited by 21
  25. 2021MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modelingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 30; 2025: 17; 2026: 13.

75 papers.

Sort:

By area:

  1. 2024Exploring scalable medical image encoders beyond text supervisioncited by 132
  2. 2023CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-trainingcited by 82
  3. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  4. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  5. 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understandingcited by 54
  6. 2023Generative Text-Guided 3D Vision-Language Pretraining for Unified Medical Image Segmentationcited by 47
  7. 2023CLIP in Medical Imaging: A Surveycited by 40
  8. 2024CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classificationcited by 40
  9. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 35
  10. 2023Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learningcited by 34
  11. 2024Masked Image Modeling: A Surveycited by 29
  12. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
38 more
  1. 2023Medical Vision Language Pretraining: A surveycited by 24
  2. 2023IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-trainingcited by 23
  3. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-trainingcited by 22
  4. 2024Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Modelscited by 18
  5. 2024Anatomical Structure-Guided Medical Vision-Language Pre-trainingcited by 18
  6. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 18
  7. 2023Label-Efficient Deep Learning in Medical Image Analysis: Challenges and Future Directionscited by 16
  8. 2023Cross-Modal Causal Intervention for Medical Report Generationcited by 15
  9. 2024CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classificationcited by 15
  10. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  11. 2025Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-trainingcited by 14
  12. 2024Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-trainingcited by 13
  13. 2023Segment Together: A Versatile Paradigm for Semi-Supervised Medical Image Segmentationcited by 12
  14. 2023G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-trainingcited by 12
  15. 2024A foundation model for generalizable disease diagnosis in chest X-ray imagescited by 12
  16. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  17. 2025MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representationscited by 11
  18. 2024PairAug: What Can Augmented Image-Text Pairs Do for Radiology?cited by 10
  19. 2024UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modelingcited by 10
  20. 2025MedMoE: Modality-Specialized Mixture of Experts for Medical Vision-Language Understandingcited by 10
  21. 2026Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learningcited by 10
  22. 2024Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammographycited by 9
  23. 2024MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-trainingcited by 8
  24. 2024MLIP: Medical Language-Image Pre-training with Masked Local Representation Learningcited by 7
  25. 2024A multi-modal vision-language model for generalizable annotation-free pathology localizationcited by 7
  26. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  27. 2024BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rayscited by 6
  28. 2025Applications of Large Models in Medicinecited by 6
  29. 2026A unified multi-task framework enables interpretable chest radiograph analysiscited by 6
  30. 2024Enhancing the vision-language foundation model with key semantic knowledge-emphasized report refinementcited by 5
  31. 2025Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Modelscited by 5
  32. 2025Data Scaling Laws for Radiology Foundation Modelscited by 5
  33. 2024Multimodal self-supervised learning for lesion localizationcited by 4
  34. 2024Enhancing medical vision-language contrastive learning via inter-matching relation modellingcited by 4
  35. 2025RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capabilitycited by 4
  36. 2023UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Promptscited by 3
  37. 2024Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniquescited by 3
  38. 2024CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuningcited by 3

50 most cited of 75 All 75 (JSON)