# PatTree: a novel approach for automated creation of multimodal, graph-based patient representations for medical classification tasks

arXiv: https://arxiv.org/abs/2608.02692 · HTML: https://mlsanity.com/paper/2608.02692/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 73 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [FastSurfer -- A fast and accurate deep learning based neuroimaging pipeline](https://mlsanity.com/paper/1910.03866/) (2019)
- [Learning the Graphical Structure of Electronic Health Records with Graph Convolutional Transformer](https://mlsanity.com/paper/1906.04716/) (2019)
- [word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings of Structured Data](https://mlsanity.com/paper/2003.12590/) (2020)
- [A generalizable 3D framework and model for self-supervised learning in medical imaging](https://mlsanity.com/paper/2501.11755/) (2025)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [Personal Health Knowledge Graph for Clinically Relevant Diet Recommendations](https://mlsanity.com/paper/2110.10131/) (2021)
- [PIPNet3D: Interpretable Detection of Alzheimer in MRI Scans](https://mlsanity.com/paper/2403.18328/) (2024)
- [Demographic Aware Probabilistic Medical Knowledge Graph Embeddings of Electronic Medical Records](https://mlsanity.com/paper/2103.11951/) (2021)
- [Towards a Personal Health Knowledge Graph Framework for Patient Monitoring](https://mlsanity.com/paper/2311.06524/) (2023)
- [MEDAKA: Construction of Biomedical Knowledge Graphs Using Large Language Models](https://mlsanity.com/paper/2509.26128/) (2025)

## Abstract (arXiv)

Access to holistic, multimodal data improves the performance of Artificial Intelligence (AI) in medical classification tasks compared to utilizing single modalities or data sources. However, the inherent heterogeneity and complexity of clinical real-world data pose significant challenges to structured data analysis and AI application. This heterogeneity includes missing values, multiple time points, diverse modalities, and inconsistent formats and semantics. Data harmonization prior to data integration tackles this challenge but remains resource-intensive and error-prone, limiting the scalability and reproducibility of holistic, AI-driven decision support on clinical real-world data. We therefore propose PatTree, a graph-based, holistic representation of patients that can be derived from real-world clinical data through the automated structuring of multimodal clinical data. PatTree enables early-stage data integration without relying on pre-standardized inputs. While representing heterogeneous clinical data within a unified knowledge graph, PatTree preserves the semantic relationships between data elements across modalities and data sources, facilitating interoperability and machine-interpretable data access. Using a subset of the ADNI-1 cohort (n = 763), we demonstrate that classification of patients is directly feasible on PatTree reaching state-of-the-art classification performance. In the three-class classification task distinguishing Alzheimer's disease, mild cognitive impairment, and cognitively normal individuals, we achieve a balanced accuracy of 98.5% and an F1 score of 0.987 on the held-out test set. Our results show that assumption-free, automated structuring of multimodal medical data can serve as a scalable foundation for clinical AI pipelines bypassing tedious data preparation and standardization.