# Unleashing the Power of Prompt-driven Nucleus Instance Segmentation

arXiv: https://arxiv.org/abs/2311.15939 · HTML: https://mlsanity.com/paper/2311.15939/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 27 Nov 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [Cell Detection with Star-convex Polygons](https://mlsanity.com/paper/1806.03535/) (2018)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework](https://mlsanity.com/paper/2107.12746/) (2021)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [CellViT: Vision Transformers for Precise Cell Segmentation and Classification](https://mlsanity.com/paper/2306.15350/) (2023)
- [DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation](https://mlsanity.com/paper/1604.02677/) (2016)
- [Methods for Segmentation and Classification of Digital Microscopy Tissue Images](https://mlsanity.com/paper/1810.13230/) (2018)
- [Micro-Net: A unified model for segmentation of various objects in microscopy images](https://mlsanity.com/paper/1804.08145/) (2018)
- [CIA-Net: Robust Nuclei Instance Segmentation with Contour-aware Information Aggregation](https://mlsanity.com/paper/1903.05358/) (2019)
- [Beyond Adapting SAM: Towards End-to-End Ultrasound Image Segmentation via Auto Prompting](https://mlsanity.com/paper/2309.06824/) (2023)
- [CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation](https://mlsanity.com/paper/2102.06867/) (2021)
- [Structure Embedded Nucleus Classification for Histopathology Images](https://mlsanity.com/paper/2302.11416/) (2023)
- [SPPNet: A Single-Point Prompt Network for Nuclei Image Segmentation](https://mlsanity.com/paper/2308.12231/) (2023)
- [PointNu-Net: Keypoint-assisted Convolutional Neural Network for Simultaneous Multi-tissue Histology Nuclei Segmentation and Classification](https://mlsanity.com/paper/2111.01557/) (2021)
- [Enhancing Nucleus Segmentation with HARU-Net: A Hybrid Attention Based Residual U-Blocks Network](https://mlsanity.com/paper/2308.03382/) (2023)

## Built on by (most cited first, 19 of 19)

- [Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation](https://mlsanity.com/paper/2401.13220/) (2024)
- [UN-SAM: Universal Prompt-Free Segmentation for Generalized Nuclei Images](https://mlsanity.com/paper/2402.16663/) (2024)
- [PLUTO: Pathology-Universal Transformer](https://mlsanity.com/paper/2405.07905/) (2024)
- [ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding](https://mlsanity.com/paper/2407.19435/) (2024)
- [De-LightSAM: Modality-Decoupled Lightweight SAM for Generalizable Medical Segmentation](https://mlsanity.com/paper/2407.14153/) (2024)
- [NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentation](https://mlsanity.com/paper/2408.11787/) (2024)
- [Towards Effective and Efficient Context-aware Nucleus Detection in Histopathology Whole Slide Images](https://mlsanity.com/paper/2503.05678/) (2025)
- [Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention](https://mlsanity.com/paper/2506.18335/) (2025)
- [Few-Shot Learning for Annotation-Efficient Nucleus Instance Segmentation](https://mlsanity.com/paper/2402.16280/) (2024)
- [Co-Seg++: Mutual Prompt-Guided Collaborative Learning for Versatile Medical Segmentation](https://mlsanity.com/paper/2506.17159/) (2025)
- [Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentation](https://mlsanity.com/paper/2509.06740/) (2025)
- [Histo-Miner: Deep learning based tissue features extraction pipeline from H&E whole slide images of cutaneous squamous cell carcinoma](https://mlsanity.com/paper/2505.04672/) (2025)
- [SAMPO-Path: Segmentation Intent-Aligned Preference Optimization for Pathology Foundation Model Segmentation](https://mlsanity.com/paper/2508.02464/) (2025)
- [Supervise Less, See More: Training-free Nuclear Instance Segmentation with Prototype-Guided Prompting](https://mlsanity.com/paper/2511.19953/) (2025)
- [NuNext: Reframing Nucleus Detection as Next-Point Detection](https://mlsanity.com/paper/2603.07098/) (2026)
- [DeNuC: Decoupling Nuclei Detection and Classification in Histopathology](https://mlsanity.com/paper/2603.04240/) (2026)
- [Instance Migration Diffusion for Nuclear Instance Segmentation in Pathology](https://mlsanity.com/paper/2504.01577/) (2025)
- [APSeg: Auto-Prompt Model with Acquired and Injected Knowledge for Nuclear Instance Segmentation and Classification](https://mlsanity.com/paper/2504.02222/) (2025)
- [Rethinking the Adaptation of Vision Foundation Models for Efficient Cell Segmentation](https://mlsanity.com/paper/2606.21913/) (2026)

## Abstract (arXiv)

Nucleus instance segmentation in histology images is crucial for a broad spectrum of clinical applications. Current dominant algorithms rely on regression of nuclear proxy maps. Distinguishing nucleus instances from the estimated maps requires carefully curated post-processing, which is error-prone and parameter-sensitive. Recently, the Segment Anything Model (SAM) has earned huge attention in medical image segmentation, owing to its impressive generalization ability and promptable property. Nevertheless, its potential on nucleus instance segmentation remains largely underexplored. In this paper, we present a novel prompt-driven framework that consists of a nucleus prompter and SAM for automatic nucleus instance segmentation. Specifically, the prompter learns to generate a unique point prompt for each nucleus while the SAM is fine-tuned to output the corresponding mask for the prompted nucleus. Furthermore, we propose the inclusion of adjacent nuclei as negative prompts to enhance the model's capability to identify overlapping nuclei. Without complicated post-processing, our proposed method sets a new state-of-the-art performance on three challenging benchmarks. Code is available at urlgithub.com/windygoo/PromptNucSeg