# PointNu-Net: Keypoint-assisted Convolutional Neural Network for Simultaneous Multi-tissue Histology Nuclei Segmentation and Classification

arXiv: https://arxiv.org/abs/2111.01557 · HTML: https://mlsanity.com/paper/2111.01557/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 1 Nov 2021.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [SOLO: Segmenting Objects by Locations](https://mlsanity.com/paper/1912.04488/) (2019)
- [RepPoints: Point Set Representation for Object Detection](https://mlsanity.com/paper/1904.11490/) (2019)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [Why rankings of biomedical image analysis competitions should be interpreted with care](https://mlsanity.com/paper/1806.02051/) (2018)
- [AdaptIS: Adaptive Instance Selection Network](https://mlsanity.com/paper/1909.07829/) (2019)
- [Micro-Net: A unified model for segmentation of various objects in microscopy images](https://mlsanity.com/paper/1804.08145/) (2018)
- [CIA-Net: Robust Nuclei Instance Segmentation with Contour-aware Information Aggregation](https://mlsanity.com/paper/1903.05358/) (2019)
- [Detecting Cancer Metastases on Gigapixel Pathology Images](https://mlsanity.com/paper/1703.02442/) (2017)
- [CentripetalNet: Pursuing High-quality Keypoint Pairs for Object Detection](https://mlsanity.com/paper/2003.09119/) (2020)
- [CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation](https://mlsanity.com/paper/2102.06867/) (2021)
- [SaccadeNet: A Fast and Accurate Object Detector](https://mlsanity.com/paper/2003.12125/) (2020)
- [Artificial neural networks in action for an automated cell-type classification of biological neural networks](https://mlsanity.com/paper/1911.09977/) (2019)

## Built on by (most cited first, 8 of 8)

- [Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation](https://mlsanity.com/paper/2401.13220/) (2024)
- [Unleashing the Power of Prompt-driven Nucleus Instance Segmentation](https://mlsanity.com/paper/2311.15939/) (2023)
- [NuNext: Reframing Nucleus Detection as Next-Point Detection](https://mlsanity.com/paper/2603.07098/) (2026)
- [NucleiMix: Realistic Data Augmentation for Nuclei Instance Segmentation](https://mlsanity.com/paper/2410.16671/) (2024)
- [DeNuC: Decoupling Nuclei Detection and Classification in Histopathology](https://mlsanity.com/paper/2603.04240/) (2026)
- [CellVTA: Enhancing Vision Foundation Models for Accurate Cell Segmentation and Classification](https://mlsanity.com/paper/2504.00784/) (2025)
- [APSeg: Auto-Prompt Model with Acquired and Injected Knowledge for Nuclear Instance Segmentation and Classification](https://mlsanity.com/paper/2504.02222/) (2025)
- [Adapting SAM to Nuclei Instance Segmentation and Classification via Cooperative Fine-Grained Refinement](https://mlsanity.com/paper/2603.28027/) (2026)

## Abstract (arXiv)

Automatic nuclei segmentation and classification play a vital role in digital pathology. However, previous works are mostly built on data with limited diversity and small sizes, making the results questionable or misleading in actual downstream tasks. In this paper, we aim to build a reliable and robust method capable of dealing with data from the 'the clinical wild'. Specifically, we study and design a new method to simultaneously detect, segment, and classify nuclei from Haematoxylin and Eosin (H&E) stained histopathology data, and evaluate our approach using the recent largest dataset: PanNuke. We address the detection and classification of each nuclei as a novel semantic keypoint estimation problem to determine the center point of each nuclei. Next, the corresponding class-agnostic masks for nuclei center points are obtained using dynamic instance segmentation. Meanwhile, we proposed a novel Joint Pyramid Fusion Module (JPFM) to model the cross-scale dependencies, thus enhancing the local feature for better nuclei detection and classification. By decoupling two simultaneous challenging tasks and taking advantage of JPFM, our method can benefit from class-aware detection and class-agnostic segmentation, thus leading to a significant performance boost. We demonstrate the superior performance of our proposed approach for nuclei segmentation and classification across 19 different tissue types, delivering new benchmark results.