# Dense Feature Learning via Linear Structure Preservation in Medical Data

arXiv: https://arxiv.org/abs/2602.07706 · HTML: https://mlsanity.com/paper/2602.07706/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 28 of 28)

- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters](https://mlsanity.com/paper/2308.08469/) (2023)
- [Machine Learning for Survival Analysis: A Survey](https://mlsanity.com/paper/1708.04649/) (2017)
- [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://mlsanity.com/paper/2307.02028/) (2023)
- [Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data](https://mlsanity.com/paper/2001.05295/) (2020)
- [MOTOR: A Time-To-Event Foundation Model For Structured Medical Records](https://mlsanity.com/paper/2301.03150/) (2023)
- [BioMegatron: Larger Biomedical Domain Language Model](https://mlsanity.com/paper/2010.06060/) (2020)
- [Multimodal LLMs for health grounded in individual-specific data](https://mlsanity.com/paper/2307.09018/) (2023)
- [Diagnosing failures of fairness transfer across distribution shift in real-world medical settings](https://mlsanity.com/paper/2202.01034/) (2022)
- [Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs](https://mlsanity.com/paper/2412.16178/) (2024)
- [TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Records](https://mlsanity.com/paper/2503.04176/) (2025)
- [EHRMamba: Towards Generalizable and Scalable Foundation Models for Electronic Health Records](https://mlsanity.com/paper/2405.14567/) (2024)
- [Clinical ModernBERT: An efficient and long context encoder for biomedical text](https://mlsanity.com/paper/2504.03964/) (2025)
- [Emergency Department Decision Support using Clinical Pseudo-notes](https://mlsanity.com/paper/2402.00160/) (2024)
- [Extremely Dense Point Correspondences using a Learned Feature Descriptor](https://mlsanity.com/paper/2003.00619/) (2020)
- [Can Large Language Models abstract Medical Coded Language?](https://mlsanity.com/paper/2403.10822/) (2024)
- [CORE-BEHRT: A Carefully Optimized and Rigorously Evaluated BEHRT](https://mlsanity.com/paper/2404.15201/) (2024)
- [HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series](https://mlsanity.com/paper/2510.25785/) (2025)
- [Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learning](https://mlsanity.com/paper/2406.13846/) (2024)
- [A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMs](https://mlsanity.com/paper/2504.14657/) (2025)
- [Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models](https://mlsanity.com/paper/2507.08254/) (2025)
- [FEET: A Framework for Evaluating Embedding Techniques](https://mlsanity.com/paper/2411.01322/) (2024)
- [Scale-Adaptive Neural Dense Features: Learning via Hierarchical Context Aggregation](https://mlsanity.com/paper/1903.10427/) (2019)
- [LabTOP: A Unified Model for Lab Test Outcome Prediction on Electronic Health Records](https://mlsanity.com/paper/2502.14259/) (2025)
- [Reflections from Research Roundtables at the Conference on Health, Inference, and Learning (CHIL) 2025](https://mlsanity.com/paper/2510.15217/) (2025)

## Abstract (arXiv)

Deep learning models for medical data are typically trained using task specific objectives that encourage representations to collapse onto a small number of discriminative directions. While effective for individual prediction problems, this paradigm underutilizes the rich structure of clinical data and limits the transferability, stability, and interpretability of learned features. In this work, we propose dense feature learning, a representation centric framework that explicitly shapes the linear structure of medical embeddings. Our approach operates directly on embedding matrices, encouraging spectral balance, subspace consistency, and feature orthogonality through objectives defined entirely in terms of linear algebraic properties. Without relying on labels or generative reconstruction, dense feature learning produces representations with higher effective rank, improved conditioning, and greater stability across time. Empirical evaluations across longitudinal EHR data, clinical text, and multimodal patient representations demonstrate consistent improvements in downstream linear performance, robustness, and subspace alignment compared to supervised and self supervised baselines. These results suggest that learning to span clinical variation may be as important as learning to predict clinical outcomes, and position representation geometry as a first class objective in medical AI.