# InstanSeg: an embedding-based instance segmentation algorithm optimized for accurate, efficient and portable cell segmentation

arXiv: https://arxiv.org/abs/2408.15954 · HTML: https://mlsanity.com/paper/2408.15954/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 28 Aug 2024.

## Builds on (most cited first, 6 of 6)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Cell Detection with Star-convex Polygons](https://mlsanity.com/paper/1806.03535/) (2018)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [NuInsSeg: A Fully Annotated Dataset for Nuclei Instance Segmentation in H&E-Stained Histological Images](https://mlsanity.com/paper/2308.01760/) (2023)
- [Instance Segmentation of Biological Images Using Harmonic Embeddings](https://mlsanity.com/paper/1904.05257/) (2019)

## Built on by (most cited first, 9 of 9)

- [Segment Anything for Histopathology](https://mlsanity.com/paper/2502.00408/) (2025)
- [Cell Instance Segmentation: The Devil Is in the Boundaries](https://mlsanity.com/paper/2510.09848/) (2025)
- [A Lightweight and Extensible Cell Segmentation and Classification Model for Whole Slide Images](https://mlsanity.com/paper/2502.19217/) (2025)
- [Towards High-Quality Image Segmentation: Improving Topology Accuracy by Penalizing Neighbor Pixels](https://mlsanity.com/paper/2603.18671/) (2026)
- [CASC-AI: Consensus-aware Self-corrective Learning for Noise Cell Segmentation](https://mlsanity.com/paper/2502.07302/) (2025)
- [Single Tensor Cell Segmentation using Scalar Field Representations](https://mlsanity.com/paper/2511.13947/) (2025)
- [CellGenNet: A Knowledge-Distilled Framework for Robust Cell Segmentation in Cancer Tissues](https://mlsanity.com/paper/2511.15054/) (2025)
- [CodeCytos: AI-assisted spatial molecular imaging analysis via code-augmented agent action space](https://mlsanity.com/paper/2606.00472/) (2026)
- [Performance at What Cost? A Sustainability-Aware Performance Index for Cell and Nucleus Instance Segmentation](https://mlsanity.com/paper/2610.10324/) (2026)

## Abstract (arXiv)

Cell and nucleus segmentation are fundamental tasks for quantitative bioimage analysis. Despite progress in recent years, biologists and other domain experts still require novel algorithms to handle increasingly large and complex real-world datasets. These algorithms must not only achieve state-of-the-art accuracy, but also be optimized for efficiency, portability and user-friendliness. Here, we introduce InstanSeg: a novel embedding-based instance segmentation pipeline designed to identify cells and nuclei in microscopy images. Using six public cell segmentation datasets, we demonstrate that InstanSeg can significantly improve accuracy when compared to the most widely used alternative methods, while reducing the processing time by at least 60%. Furthermore, InstanSeg is designed to be fully serializable as TorchScript and supports GPU acceleration on a range of hardware. We provide an open-source implementation of InstanSeg in Python, in addition to a user-friendly, interactive QuPath extension for inference written in Java. Our code and pre-trained models are available at https://github.com/instanseg/instanseg .