# DoNet: Deep De-overlapping Network for Cytology Instance Segmentation

arXiv: https://arxiv.org/abs/2303.14373 · HTML: https://mlsanity.com/paper/2303.14373/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 25 Mar 2023.

## Builds on (most cited first, 17 of 17)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Mask Scoring R-CNN](https://mlsanity.com/paper/1903.00241/) (2019)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation](https://mlsanity.com/paper/1604.02677/) (2016)
- [Amodal Instance Segmentation](https://mlsanity.com/paper/1604.08202/) (2016)
- [Learning to See the Invisible: End-to-End Trainable Amodal Instance Segmentation](https://mlsanity.com/paper/1804.08864/) (2018)
- [Amodal Segmentation Based on Visible Region Segmentation and Shape Prior](https://mlsanity.com/paper/2012.05598/) (2020)
- [Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling](https://mlsanity.com/paper/2109.11103/) (2021)
- [Deep Semi-supervised Knowledge Distillation for Overlapping Cervical Cell Instance Segmentation](https://mlsanity.com/paper/2007.10787/) (2020)
- [A Weakly Supervised Amodal Segmenter with Boundary Uncertainty Estimation](https://mlsanity.com/paper/2108.09897/) (2021)
- [Deep Learning for Computational Cytology: A Survey](https://mlsanity.com/paper/2202.05126/) (2022)
- [Learning Robust Representation for Joint Grading of Ophthalmic Diseases via Adaptive Curriculum and Feature Disentanglement](https://mlsanity.com/paper/2207.04183/) (2022)
- [IRNet: Instance Relation Network for Overlapping Cervical Cell Segmentation](https://mlsanity.com/paper/1908.06623/) (2019)
- [MultiStar: Instance Segmentation of Overlapping Objects with Star-Convex Polygons](https://mlsanity.com/paper/2011.13228/) (2020)
- [Deep Semi-supervised Metric Learning with Dual Alignment for Cervical Cancer Cell Detection](https://mlsanity.com/paper/2104.03265/) (2021)

## Built on by (most cited first, 17 of 17)

- [DAWN: Domain-Adaptive Weakly Supervised Nuclei Segmentation via Cross-Task Interactions](https://mlsanity.com/paper/2404.14956/) (2024)
- [SEINE: Structure Encoding and Interaction Network for Nuclei Instance Segmentation](https://mlsanity.com/paper/2401.09773/) (2024)
- [Inter- and intra-uncertainty based feature aggregation model for semi-supervised histopathology image segmentation](https://mlsanity.com/paper/2403.12767/) (2024)
- [Category Prompt Mamba Network for Nuclei Segmentation and Classification](https://mlsanity.com/paper/2503.10422/) (2025)
- [The Four Color Theorem for Cell Instance Segmentation](https://mlsanity.com/paper/2506.09724/) (2025)
- [GAInS: Gradient Anomaly-aware Biomedical Instance Segmentation](https://mlsanity.com/paper/2409.13988/) (2024)
- [Supervise Less, See More: Training-free Nuclear Instance Segmentation with Prototype-Guided Prompting](https://mlsanity.com/paper/2511.19953/) (2025)
- [Disco: Densely-overlapping Cell Instance Segmentation via Adjacency-aware Collaborative Coloring](https://mlsanity.com/paper/2602.05420/) (2026)
- [Cyto R-CNN and CytoNuke Dataset: Towards reliable whole-cell segmentation in bright-field histological images](https://mlsanity.com/paper/2401.15638/) (2024)
- [CellMamba: Adaptive Mamba for Accurate and Efficient Cell Detection](https://mlsanity.com/paper/2512.21803/) (2025)
- [Holistic and Historical Instance Comparison for Cervical Cell Detection](https://mlsanity.com/paper/2409.13987/) (2024)
- [LSA: Latent Style Augmentation Towards Stain-Agnostic Cervical Cancer Screening](https://mlsanity.com/paper/2503.06563/) (2025)
- [Overlap-aware segmentation for topological reconstruction of obscured objects](https://mlsanity.com/paper/2510.06194/) (2025)
- [Weak-to-Strong Generalization Enables Fully Automated De Novo Training of Multi-head Mask-RCNN Model for Segmenting Densely Overlapping Cell Nuclei in Multiplex Whole-slice Brain Images](https://mlsanity.com/paper/2512.11722/) (2025)
- [QCell: Recombining and Aligning Cell Queries for Overlapping Instance Segmentation](https://mlsanity.com/paper/2608.29253/) (2026)
- [End-to-End Cell Detection via Instance-aware Graph Modeling](https://mlsanity.com/paper/2609.15354/) (2026)
- [One Tile, Multiple Instances: Rethinking MIL for Sparse Diagnostic Evidence](https://mlsanity.com/paper/2610.04853/) (2026)

## Abstract (arXiv)

Cell instance segmentation in cytology images has significant importance for biology analysis and cancer screening, while remains challenging due to 1) the extensive overlapping translucent cell clusters that cause the ambiguous boundaries, and 2) the confusion of mimics and debris as nuclei. In this work, we proposed a De-overlapping Network (DoNet) in a decompose-and-recombined strategy. A Dual-path Region Segmentation Module (DRM) explicitly decomposes the cell clusters into intersection and complement regions, followed by a Semantic Consistency-guided Recombination Module (CRM) for integration. To further introduce the containment relationship of the nucleus in the cytoplasm, we design a Mask-guided Region Proposal Strategy (MRP) that integrates the cell attention maps for inner-cell instance prediction. We validate the proposed approach on ISBI2014 and CPS datasets. Experiments show that our proposed DoNet significantly outperforms other state-of-the-art (SOTA) cell instance segmentation methods. The code is available at https://github.com/DeepDoNet/DoNet.