# APSeg: Auto-Prompt Model with Acquired and Injected Knowledge for Nuclear Instance Segmentation and Classification

arXiv: https://arxiv.org/abs/2504.02222 · HTML: https://mlsanity.com/paper/2504.02222/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [CellViT: Vision Transformers for Precise Cell Segmentation and Classification](https://mlsanity.com/paper/2306.15350/) (2023)
- [SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation](https://mlsanity.com/paper/2308.08746/) (2023)
- [SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation](https://mlsanity.com/paper/2408.08870/) (2024)
- [Input Augmentation with SAM: Boosting Medical Image Segmentation with Segmentation Foundation Model](https://mlsanity.com/paper/2304.11332/) (2023)
- [Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding](https://mlsanity.com/paper/2403.18271/) (2024)
- [VLCounter: Text-aware Visual Representation for Zero-Shot Object Counting](https://mlsanity.com/paper/2312.16580/) (2023)
- [Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation](https://mlsanity.com/paper/2401.13220/) (2024)
- [Unleashing the Power of Prompt-driven Nucleus Instance Segmentation](https://mlsanity.com/paper/2311.15939/) (2023)
- [TP-DRSeg: Improving Diabetic Retinopathy Lesion Segmentation with Explicit Text-Prompts Assisted SAM](https://mlsanity.com/paper/2406.15764/) (2024)
- [ASPS: Augmented Segment Anything Model for Polyp Segmentation](https://mlsanity.com/paper/2407.00718/) (2024)
- [PointNu-Net: Keypoint-assisted Convolutional Neural Network for Simultaneous Multi-tissue Histology Nuclei Segmentation and Classification](https://mlsanity.com/paper/2111.01557/) (2021)
- [DB-SAM: Delving into High Quality Universal Medical Image Segmentation](https://mlsanity.com/paper/2410.04172/) (2024)
- [Mask-Enhanced Segment Anything Model for Tumor Lesion Semantic Segmentation](https://mlsanity.com/paper/2403.05912/) (2024)

## Abstract (arXiv)

Nuclear instance segmentation and classification provide critical quantitative foundations for digital pathology diagnosis. With the advent of the foundational Segment Anything Model (SAM), the accuracy and efficiency of nuclear segmentation have improved significantly. However, SAM imposes a strong reliance on precise prompts, and its class-agnostic design renders its classification results entirely dependent on the provided prompts. Therefore, we focus on generating prompts with more accurate localization and classification and propose APSeg, Auto-Prompt model with acquired and injected knowledge for nuclear instance Segmentation and classification. APSeg incorporates two knowledge-aware modules: (1) Distribution-Guided Proposal Offset Module (DG-POM), which learns distribution knowledge through density map guided, and (2) Category Knowledge Semantic Injection Module (CK-SIM), which injects morphological knowledge derived from category descriptions. We conducted extensive experiments on the PanNuke and CoNSeP datasets, demonstrating the effectiveness of our approach. The code will be released upon acceptance.