# FluoroSAM: A Language-promptable Foundation Model for Flexible X-ray Image Segmentation

arXiv: https://arxiv.org/abs/2403.08059 · HTML: https://mlsanity.com/paper/2403.08059/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Improving CLIP Training with Language Rewrites](https://mlsanity.com/paper/2305.20088/) (2023)
- [EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention](https://mlsanity.com/paper/2305.07027/) (2023)
- [Deep Learning for Chest X-ray Analysis: A Survey](https://mlsanity.com/paper/2103.08700/) (2021)
- [DeepDRR -- A Catalyst for Machine Learning in Fluoroscopy-guided Procedures](https://mlsanity.com/paper/1803.08606/) (2018)
- [Automatic segmenting teeth in X-ray images: Trends, a novel data set, benchmarking and future perspectives](https://mlsanity.com/paper/1802.03086/) (2018)
- [Pelphix: Surgical Phase Recognition from X-ray Images in Percutaneous Pelvic Fixation](https://mlsanity.com/paper/2304.09285/) (2023)

## Built on by (most cited first, 7 of 7)

- [Intelligent Control of Robotic X-ray Devices using a Language-promptable Digital Twin](https://mlsanity.com/paper/2412.08020/) (2024)
- [Investigating Robot Control Policy Learning for Autonomous X-ray-guided Spine Procedures](https://mlsanity.com/paper/2511.03882/) (2025)
- [TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research](https://mlsanity.com/paper/2511.07412/) (2025)
- [Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation](https://mlsanity.com/paper/2606.17340/) (2026)
- [2D Versus 3D Diffusion for In Silico Training of Interventional X-ray AI Models](https://mlsanity.com/paper/2606.21414/) (2026)
- [Patient-Specific Articulated Digital Twins from a Single Full-Body CT Scan](https://mlsanity.com/paper/2607.02156/) (2026)
- [FleXray: Universal Clinical X-ray Segmentation](https://mlsanity.com/paper/2609.26756/) (2026)

## Abstract (arXiv)

Language promptable X-ray image segmentation would enable greater flexibility for human-in-the-loop workflows in diagnostic and interventional precision medicine. Prior efforts have contributed task-specific models capable of solving problems within a narrow scope, but expanding to broader use requires additional data, annotations, and training time. Recently, language-aligned foundation models (LFMs) -- machine learning models trained on large amounts of highly variable image and text data thus enabling broad applicability -- have emerged as promising tools for automated image analysis. Existing foundation models for medical image analysis focus on scenarios and modalities where large, richly annotated datasets are available. However, the X-ray imaging modality features highly variable image appearance and applications, from diagnostic chest X-rays to interventional fluoroscopy, with varying availability of data. To pave the way toward an LFM for comprehensive and language-aligned analysis of arbitrary medical X-ray images, we introduce FluoroSAM, a language-promptable variant of the Segment Anything Model, trained from scratch on 3M synthetic X-ray images from a wide variety of human anatomies, imaging geometries, and viewing angles. These include pseudo-ground truth masks for 128 organ types and 464 tools with associated text descriptions. FluoroSAM is capable of segmenting myriad anatomical structures and tools based on natural language prompts, thanks to the novel incorporation of vector quantization (VQ) of text embeddings in the training process. We demonstrate FluoroSAM's performance quantitatively on real X-ray images and showcase on several applications how FluoroSAM is a key enabler for rich human-machine interaction in the X-ray image acquisition and analysis context. Code is available at https://github.com/arcadelab/fluorosam.