# Fine-grained Multi-class Nuclei Segmentation with Molecular-empowered All-in-SAM Model

arXiv: https://arxiv.org/abs/2508.15751 · HTML: https://mlsanity.com/paper/2508.15751/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [How Do Vision Transformers Work?](https://mlsanity.com/paper/2202.06709/) (2022)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [Understanding The Robustness in Vision Transformers](https://mlsanity.com/paper/2204.12451/) (2022)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding](https://mlsanity.com/paper/2310.15308/) (2023)
- [CellViT: Vision Transformers for Precise Cell Segmentation and Classification](https://mlsanity.com/paper/2306.15350/) (2023)
- [Deep Learning for Whole Slide Image Analysis: An Overview](https://mlsanity.com/paper/1910.11097/) (2019)
- [Self-Supervised Nuclei Segmentation in Histopathological Images Using Attention](https://mlsanity.com/paper/2007.08373/) (2020)
- [Foundation Models for Biomedical Image Segmentation: A Survey](https://mlsanity.com/paper/2401.07654/) (2024)
- [Instance-aware Self-supervised Learning for Nuclei Segmentation](https://mlsanity.com/paper/2007.11186/) (2020)
- [Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation](https://mlsanity.com/paper/2401.13220/) (2024)
- [Computational Pathology: Challenges and Promises for Tissue Analysis](https://mlsanity.com/paper/1601.00027/) (2015)
- [Democratizing Pathological Image Segmentation with Lay Annotators via Molecular-empowered Learning](https://mlsanity.com/paper/2306.00047/) (2023)
- [Glo-In-One: Holistic Glomerular Detection, Segmentation, and Lesion Characterization with Large-scale Web Image Mining](https://mlsanity.com/paper/2206.00123/) (2022)

## Built on by (most cited first, 2 of 2)

- [From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathology](https://mlsanity.com/paper/2511.11984/) (2025)
- [Granular Computing-driven SAM: From Coarse-to-Fine Guidance for Prompt-Free Segmentation](https://mlsanity.com/paper/2511.19062/) (2025)

## Abstract (arXiv)

Purpose: Recent developments in computational pathology have been driven by advances in Vision Foundation Models, particularly the Segment Anything Model (SAM). This model facilitates nuclei segmentation through two primary methods: prompt-based zero-shot segmentation and the use of cell-specific SAM models for direct segmentation. These approaches enable effective segmentation across a range of nuclei and cells. However, general vision foundation models often face challenges with fine-grained semantic segmentation, such as identifying specific nuclei subtypes or particular cells. Approach: In this paper, we propose the molecular-empowered All-in-SAM Model to advance computational pathology by leveraging the capabilities of vision foundation models. This model incorporates a full-stack approach, focusing on: (1) annotation-engaging lay annotators through molecular-empowered learning to reduce the need for detailed pixel-level annotations, (2) learning-adapting the SAM model to emphasize specific semantics, which utilizes its strong generalizability with SAM adapter, and (3) refinement-enhancing segmentation accuracy by integrating Molecular-Oriented Corrective Learning (MOCL). Results: Experimental results from both in-house and public datasets show that the All-in-SAM model significantly improves cell classification performance, even when faced with varying annotation quality. Conclusions: Our approach not only reduces the workload for annotators but also extends the accessibility of precise biomedical image analysis to resource-limited settings, thereby advancing medical diagnostics and automating pathology image analysis.