# Health AI Developer Foundations

arXiv: https://arxiv.org/abs/2411.15128 · HTML: https://mlsanity.com/paper/2411.15128/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [A deep learning system for differential diagnosis of skin diseases](https://mlsanity.com/paper/1909.05382/) (2019)
- [Advancing Multimodal Medical Capabilities of Gemini](https://mlsanity.com/paper/2405.03162/) (2024)
- [BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2109.13226/) (2021)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners](https://mlsanity.com/paper/2212.04979/) (2022)
- [ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders](https://mlsanity.com/paper/2308.01317/) (2023)
- [HeAR -- Health Acoustic Representations](https://mlsanity.com/paper/2403.02522/) (2024)
- [FRILL: A Non-Semantic Speech Embedding for Mobile Devices](https://mlsanity.com/paper/2011.04609/) (2020)
- [Domain-specific optimization and diverse evaluation of self-supervised models for histopathology](https://mlsanity.com/paper/2310.13259/) (2023)
- [Closing the AI generalization gap by adjusting for dermatology condition distribution differences across clinical settings](https://mlsanity.com/paper/2402.15566/) (2024)

## Built on by (most cited first, 4 of 4)

- [RoentMod: A Synthetic Chest X-Ray Modification Model to Identify and Correct Image Interpretation Model Shortcuts](https://mlsanity.com/paper/2509.08640/) (2025)
- [CureAgent: A Training-Free Executor-Analyst Framework for Clinical Reasoning](https://mlsanity.com/paper/2512.05576/) (2025)
- [Picking the Right Image to Classify: Reliable-Input Selection in Teledermatology](https://mlsanity.com/paper/2608.16198/) (2026)
- [Adaptive Conformal Redistribution for Inter-class Transitional Uncertainty in Medical Image Classification](https://mlsanity.com/paper/2609.13303/) (2026)

## Abstract (arXiv)

Robust medical Machine Learning (ML) models have the potential to revolutionize healthcare by accelerating clinical research, improving workflows and outcomes, and producing novel insights or capabilities. Developing such ML models from scratch is cost prohibitive and requires substantial compute, data, and time (e.g., expert labeling). To address these challenges, we introduce Health AI Developer Foundations (HAI-DEF), a suite of pre-trained, domain-specific foundation models, tools, and recipes to accelerate building ML for health applications. The models cover various modalities and domains, including radiology (X-rays and computed tomography), histopathology, dermatological imaging, and audio. These models provide domain specific embeddings that facilitate AI development with less labeled data, shorter training times, and reduced computational costs compared to traditional approaches. In addition, we utilize a common interface and style across these models, and prioritize usability to enable developers to integrate HAI-DEF efficiently. We present model evaluations across various tasks and conclude with a discussion of their application and evaluation, covering the importance of ensuring efficacy, fairness, and equity. Finally, while HAI-DEF and specifically the foundation models lower the barrier to entry for ML in healthcare, we emphasize the importance of validation with problem- and population-specific data for each desired usage setting. This technical report will be updated over time as more modalities and features are added.