# General Demographic Pre-trained Models for Enhancing Predictive Performance Across Diseases and Population

arXiv: https://arxiv.org/abs/2509.07330 · HTML: https://mlsanity.com/paper/2509.07330/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 48 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TS2Vec: Towards Universal Representation of Time Series](https://mlsanity.com/paper/2106.10466/) (2021)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [Multimodal Whole Slide Foundation Model for Pathology](https://mlsanity.com/paper/2411.19666/) (2024)
- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [GenHPF: General Healthcare Predictive Framework with Multi-task Multi-source Learning](https://mlsanity.com/paper/2207.09858/) (2022)
- [EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records](https://mlsanity.com/paper/2405.14567/) (2024)
- [A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records](https://mlsanity.com/paper/2311.11483/) (2023)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [Self-supervised learning-based general laboratory progress pretrained model for cardiovascular event detection](https://mlsanity.com/paper/2303.06980/) (2023)
- [Investigating Convolutional Neural Networks using Spatial Orderness](https://mlsanity.com/paper/1908.06416/) (2019)

## Abstract (arXiv)

Foundation models for healthcare require balancing robust generalization across heterogeneous clinical populations and disease settings with the architectural simplicity needed for deployment. We present a pre-trained model focused on demographic attributes that enhances feature utility across medical domains in a plug-and-play fashion. We introduce the General Demographic Pre-trained (GDP) model, designed to extract intrinsic representations of patient status based on age and sex, the two most ubiquitous clinical features. The composition of GDP was optimized by investigating various encoding methods and visit-reordering schemes. The model was pre-trained and transferability was validated by embedding the learned representations into diverse disease and geographic cohorts characterized by distinct demographic profiles. The optimal model configuration was subsequently validated against top-performing tabular foundation models (TabPFN, TabICL, and TabFM). Our findings demonstrate that concatenating GDP-derived embeddings with raw residual features consistently enhances predictive performance across classification tasks while elevating the relative importance of demographic attributes. The embedding transformation provides superior representational separability compared to the original data distribution, yielding competitive discrimination performance across metrics against all three general-purpose foundation models and tree-based algorithm. GDP has successfully served the purpose of a foundation model, which produce enriched representations that amplify the predictive insight of these features beyond their raw form. The generated embeddings can be directly concatenated with residual features, serving as an enhancement layer that maintains full compatibility with standard tabular classifiers.