# A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

arXiv: https://arxiv.org/abs/2306.00864 · HTML: https://mlsanity.com/paper/2306.00864/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 58 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports](https://mlsanity.com/paper/2111.03452/) (2021)
- [Diagnose Like a Radiologist: Hybrid Neuro-Probabilistic Reasoning for Attribute-Based Medical Image Diagnosis](https://mlsanity.com/paper/2208.09282/) (2022)
- [MVP-Net: Multi-view FPN with Position-aware Attention for Deep Universal Lesion Detection](https://mlsanity.com/paper/1909.04247/) (2019)

## Built on by (most cited first, 30 of 43)

- [Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning](https://mlsanity.com/paper/2309.05904/) (2023)
- [Emergency Department Decision Support using Clinical Pseudo-notes](https://mlsanity.com/paper/2402.00160/) (2024)
- [Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models](https://mlsanity.com/paper/2401.12215/) (2024)
- [Multimodal Data Integration for Precision Oncology: Challenges and Future Directions](https://mlsanity.com/paper/2406.19611/) (2024)
- [Activate and Reject: Towards Safe Domain Generalization under Category Shift](https://mlsanity.com/paper/2310.04724/) (2023)
- [Adaptive Betweenness Clustering for Semi-Supervised Domain Adaptation](https://mlsanity.com/paper/2401.11448/) (2024)
- [Memorize and Rank: Elevating Large Language Models for Clinical Diagnosis Prediction](https://mlsanity.com/paper/2501.17326/) (2025)
- [Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning](https://mlsanity.com/paper/2509.19090/) (2025)
- [AliFuse: Aligning and Fusing Multi-modal Medical Data for Computer-Aided Diagnosis](https://mlsanity.com/paper/2401.01074/) (2024)
- [Serialized EHR make for good text representations](https://mlsanity.com/paper/2510.13843/) (2025)
- [FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Prediction](https://mlsanity.com/paper/2406.11928/) (2024)
- [Multimodal self-supervised learning for lesion localization](https://mlsanity.com/paper/2401.01524/) (2024)
- [Insulin Resistance Prediction From Wearables and Routine Blood Biomarkers](https://mlsanity.com/paper/2505.03784/) (2025)
- [Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques](https://mlsanity.com/paper/2401.01583/) (2024)
- [Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification](https://mlsanity.com/paper/2401.07796/) (2024)
- [Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism](https://mlsanity.com/paper/2406.06594/) (2024)
- [HoloDx: Knowledge- and Data-Driven Multimodal Diagnosis of Alzheimer's Disease](https://mlsanity.com/paper/2504.19075/) (2025)
- [Adapting Machine Learning Diagnostic Models to New Populations Using a Small Amount of Data: Results from Clinical Neuroscience](https://mlsanity.com/paper/2308.03175/) (2023)
- [DocParseNet: Advanced Semantic Segmentation and OCR Embeddings for Efficient Scanned Document Annotation](https://mlsanity.com/paper/2406.17591/) (2024)
- [Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning](https://mlsanity.com/paper/2407.14904/) (2024)
- [Can Physician Judgment Enhance Model Trustworthiness? A Case Study on Predicting Pathological Lymph Nodes in Rectal Cancer](https://mlsanity.com/paper/2312.09529/) (2023)
- [Prospects for AI-Enhanced ECG as a Unified Screening Tool for Cardiac and Non-Cardiac Conditions -- An Explorative Study in Emergency Care](https://mlsanity.com/paper/2312.11050/) (2023)
- [Multimodal Variational Autoencoder for Low-cost Cardiac Hemodynamics Instability Detection](https://mlsanity.com/paper/2403.13658/) (2024)
- [Investigating the validity of structure learning algorithms in identifying risk factors for intervention in patients with diabetes](https://mlsanity.com/paper/2403.14327/) (2024)
- [A Survey for Large Language Models in Biomedicine](https://mlsanity.com/paper/2409.00133/) (2024)
- [A Medical Multimodal Large Language Model for Pediatric Pneumonia](https://mlsanity.com/paper/2409.02608/) (2024)
- [HistoEncoder: a digital pathology foundation model for prostate cancer](https://mlsanity.com/paper/2411.11458/) (2024)
- [DAFTED: Decoupled Asymmetric Fusion of Tabular and Echocardiographic Data for Cardiac Hypertension Diagnosis](https://mlsanity.com/paper/2509.15990/) (2025)
- [Language-Enhanced Generative Modeling for Amyloid PET Synthesis from MRI and Blood Biomarkers](https://mlsanity.com/paper/2511.02206/) (2025)
- [LeMoF: Level-guided Multimodal Fusion for Heterogeneous Clinical Data](https://mlsanity.com/paper/2601.10092/) (2026)

## Abstract (arXiv)

During the diagnostic process, clinicians leverage multimodal information, such as chief complaints, medical images, and laboratory-test results. Deep-learning models for aiding diagnosis have yet to meet this requirement. Here we report a Transformer-based representation-learning model as a clinical diagnostic aid that processes multimodal input in a unified manner. Rather than learning modality-specific features, the model uses embedding layers to convert images and unstructured and structured text into visual tokens and text tokens, and bidirectional blocks with intramodal and intermodal attention to learn a holistic representation of radiographs, the unstructured chief complaint and clinical history, structured clinical information such as laboratory-test results and patient demographic information. The unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of pulmonary diseases (by 12% and 9%, respectively) and in the prediction of adverse clinical outcomes in patients with COVID-19 (by 29% and 7%, respectively). Leveraging unified multimodal Transformer-based models may help streamline triage of patients and facilitate the clinical decision process.