# CellPilot: A unified approach to automatic and interactive segmentation in histopathology

arXiv: https://arxiv.org/abs/2411.15514 · HTML: https://mlsanity.com/paper/2411.15514/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [Customized Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.13785/) (2023)
- [Segment Anything in Medical Images](https://mlsanity.com/paper/2304.12306/) (2023)
- [Gland Segmentation in Colon Histology Images: The GlaS Challenge Contest](https://mlsanity.com/paper/1603.00275/) (2016)
- [The Road Less Scheduled](https://mlsanity.com/paper/2405.15682/) (2024)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [SimpleClick: Interactive Image Segmentation with Simple Vision Transformers](https://mlsanity.com/paper/2210.11006/) (2022)
- [CellViT: Vision Transformers for Precise Cell Segmentation and Classification](https://mlsanity.com/paper/2306.15350/) (2023)
- [MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Images](https://mlsanity.com/paper/1806.01963/) (2018)
- [Methods for Segmentation and Classification of Digital Microscopy Tissue Images](https://mlsanity.com/paper/1810.13230/) (2018)
- [CryoNuSeg: A Dataset for Nuclei Instance Segmentation of Cryosectioned H&E-Stained Histological Images](https://mlsanity.com/paper/2101.00442/) (2021)
- [NuClick: A Deep Learning Framework for Interactive Segmentation of Microscopy Images](https://mlsanity.com/paper/2005.14511/) (2020)
- [CoNIC: Colon Nuclei Identification and Counting Challenge 2022](https://mlsanity.com/paper/2111.14485/) (2021)
- [The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions](https://mlsanity.com/paper/2308.05864/) (2023)
- [SAM-Path: A Segment Anything Model for Semantic Segmentation in Digital Pathology](https://mlsanity.com/paper/2307.09570/) (2023)
- [Evaluation and improvement of Segment Anything Model for interactive histopathology image segmentation](https://mlsanity.com/paper/2310.10493/) (2023)

## Abstract (arXiv)

Histopathology, the microscopic study of diseased tissue, is increasingly digitized, enabling improved visualization and streamlined workflows. An important task in histopathology is the segmentation of cells and glands, essential for determining shape and frequencies that can serve as indicators of disease. Deep learning tools are widely used in histopathology. However, variability in tissue appearance and cell morphology presents challenges for achieving reliable segmentation, often requiring manual correction to improve accuracy. This work introduces CellPilot, a framework that bridges the gap between automatic and interactive segmentation by providing initial automatic segmentation as well as guided interactive refinement. Our model was trained on over 675,000 masks of nine diverse cell and gland segmentation datasets, spanning 16 organs. CellPilot demonstrates superior performance compared to other interactive tools on three held-out histopathological datasets while enabling automatic segmentation. We make the model and a graphical user interface designed to assist practitioners in creating large-scale annotated datasets available as open-source, fostering the development of more robust and generalized diagnostic models.