# A three in one bottom-up framework for simultaneous semantic segmentation, instance segmentation and classification of multi-organ nuclei in digital cancer histology

arXiv: https://arxiv.org/abs/2308.11179 · HTML: https://mlsanity.com/paper/2308.11179/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 19 of 19)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [CenterMask : Real-Time Anchor-Free Instance Segmentation](https://mlsanity.com/paper/1911.06667/) (2019)
- [PolarMask: Single Shot Instance Segmentation with Polar Representation](https://mlsanity.com/paper/1909.13226/) (2019)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [TensorMask: A Foundation for Dense Object Segmentation](https://mlsanity.com/paper/1903.12174/) (2019)
- [Methods for Segmentation and Classification of Digital Microscopy Tissue Images](https://mlsanity.com/paper/1810.13230/) (2018)
- [Micro-Net: A unified model for segmentation of various objects in microscopy images](https://mlsanity.com/paper/1804.08145/) (2018)
- [MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features](https://mlsanity.com/paper/1712.04837/) (2017)
- [CIA-Net: Robust Nuclei Instance Segmentation with Contour-aware Information Aggregation](https://mlsanity.com/paper/1903.05358/) (2019)
- [Detecting Cancer Metastases on Gigapixel Pathology Images](https://mlsanity.com/paper/1703.02442/) (2017)
- [CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection](https://mlsanity.com/paper/2003.09119/) (2020)
- [One Model is All You Need: Multi-Task Learning Enables Simultaneous Histology Image Segmentation and Classification](https://mlsanity.com/paper/2203.00077/) (2022)
- [SaccadeNet: A Fast and Accurate Object Detector](https://mlsanity.com/paper/2003.12125/) (2020)

## Abstract (arXiv)

Simultaneous segmentation and classification of nuclei in digital histology play an essential role in computer-assisted cancer diagnosis; however, it remains challenging. The highest achieved binary and multi-class Panoptic Quality (PQ) remains as low as 0.68 bPQ and 0.49 mPQ, respectively. It is due to the higher staining variability, variability across the tissue, rough clinical conditions, overlapping nuclei, and nuclear class imbalance. The generic deep-learning methods usually rely on end-to-end models, which fail to address these problems associated explicitly with digital histology. In our previous work, DAN-NucNet, we resolved these issues for semantic segmentation with an end-to-end model. This work extends our previous model to simultaneous instance segmentation and classification. We introduce additional decoder heads with independent weighted losses, which produce semantic segmentation, edge proposals, and classification maps. We use the outputs from the three-head model to apply post-processing to produce the final segmentation and classification. Our multi-stage approach utilizes edge proposals and semantic segmentations compared to direct segmentation and classification strategies followed by most state-of-the-art methods. Due to this, we demonstrate a significant performance improvement in producing high-quality instance segmentation and nuclei classification. We have achieved a 0.841 Dice score for semantic segmentation, 0.713 bPQ scores for instance segmentation, and 0.633 mPQ for nuclei classification. Our proposed framework is generalized across 19 types of tissues. Furthermore, the framework is less complex compared to the state-of-the-art.