# Omni-Seg: A Scale-aware Dynamic Network for Renal Pathological Image Segmentation

arXiv: https://arxiv.org/abs/2206.13632 · HTML: https://mlsanity.com/paper/2206.13632/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 55 references that are in the map.
- First posted on arXiv 27 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [CE-Net: Context Encoder Network for 2D Medical Image Segmentation](https://mlsanity.com/paper/1903.02740/) (2019)
- [DoDNet: Learning to segment multi-organ and tumors from multiple partially labeled datasets](https://mlsanity.com/paper/2011.10217/) (2020)
- [Multi-scale self-guided attention for medical image segmentation](https://mlsanity.com/paper/1906.02849/) (2019)
- [Joint Optic Disc and Cup Segmentation Based on Multi-label Deep Network and Polar Transformation](https://mlsanity.com/paper/1801.00926/) (2018)
- [Multi-organ Segmentation over Partially Labeled Datasets with Multi-scale Feature Abstraction](https://mlsanity.com/paper/2001.00208/) (2020)
- [ModDrop++: A Dynamic Filter Network with Intra-subject Co-training for Multiple Sclerosis Lesion Segmentation with Missing Modalities](https://mlsanity.com/paper/2203.04959/) (2022)
- [Iterative annotation to ease neural network training: Specialized machine learning in medical image analysis](https://mlsanity.com/paper/1812.07509/) (2018)
- [CNN Cascades for Segmenting Whole Slide Images of the Kidney](https://mlsanity.com/paper/1708.00251/) (2017)
- [Evaluating Transformer-based Semantic Segmentation Networks for Pathological Image Segmentation](https://mlsanity.com/paper/2108.11993/) (2021)
- [Pyramid U-Net for Retinal Vessel Segmentation](https://mlsanity.com/paper/2104.02333/) (2021)
- [Omni-Seg: A Single Dynamic Network for Multi-label Renal Pathology Image Segmentation using Partially Labeled Data](https://mlsanity.com/paper/2112.12665/) (2021)

## Built on by (most cited first, 16 of 16)

- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [COSST: Multi-organ Segmentation with Partially Labeled Datasets Using Comprehensive Supervisions and Self-training](https://mlsanity.com/paper/2304.14030/) (2023)
- [PrPSeg: Universal Proposition Learning for Panoramic Renal Pathology Segmentation](https://mlsanity.com/paper/2402.19286/) (2024)
- [HATs: Hierarchical Adaptive Taxonomy Segmentation for Panoramic Pathology Image Analysis](https://mlsanity.com/paper/2407.00596/) (2024)
- [HoloHisto: End-to-end Gigapixel WSI Segmentation with 4K Resolution Sequential Tokenization](https://mlsanity.com/paper/2407.03307/) (2024)
- [MedUniSeg: 2D and 3D Medical Image Segmentation via a Prompt-driven Universal Model](https://mlsanity.com/paper/2410.05905/) (2024)
- [PFPs: Prompt-guided Flexible Pathological Segmentation for Diverse Potential Outcomes Using Large Vision and Language Models](https://mlsanity.com/paper/2407.09979/) (2024)
- [IRS: Incremental Relationship-guided Segmentation for Digital Pathology](https://mlsanity.com/paper/2505.22855/) (2025)
- [BackSplit: The Importance of Sub-dividing the Background in Biomedical Lesion Segmentation](https://mlsanity.com/paper/2511.19394/) (2025)
- [DyMorph-B2I: Dynamic and Morphology-Guided Binary-to-Instance Segmentation for Renal Pathology](https://mlsanity.com/paper/2508.15208/) (2025)
- [M^3-GloDets: Multi-Region and Multi-Scale Analysis of Fine-Grained Diseased Glomerular Detection](https://mlsanity.com/paper/2508.17666/) (2025)
- [NMGrad: Advancing Histopathological Bladder Cancer Grading with Weakly Supervised Deep Learning](https://mlsanity.com/paper/2405.15275/) (2024)
- [3D Cell Oversegmentation Correction via Geo-Wasserstein Divergence](https://mlsanity.com/paper/2502.01890/) (2025)
- [PathSeqSAM: Sequential Modeling for Pathology Image Segmentation with SAM2](https://mlsanity.com/paper/2504.10526/) (2025)
- [How Close Are We? Limitations and Progress of AI Models in Banff Lesion Scoring](https://mlsanity.com/paper/2510.27158/) (2025)
- [Seeing Abnormal from Normal: Glomerular Abnormality in Representations of Normal Renal Morphology](https://mlsanity.com/paper/2609.19444/) (2026)

## Abstract (arXiv)

Comprehensive semantic segmentation on renal pathological images is challenging due to the heterogeneous scales of the objects. For example, on a whole slide image (WSI), the cross-sectional areas of glomeruli can be 64 times larger than that of the peritubular capillaries, making it impractical to segment both objects on the same patch, at the same scale. To handle this scaling issue, prior studies have typically trained multiple segmentation networks in order to match the optimal pixel resolution of heterogeneous tissue types. This multi-network solution is resource-intensive and fails to model the spatial relationship between tissue types. In this paper, we propose the Omni-Seg+ network, a scale-aware dynamic neural network that achieves multi-object (six tissue types) and multi-scale (5X to 40X scale) pathological image segmentation via a single neural network. The contribution of this paper is three-fold: (1) a novel scale-aware controller is proposed to generalize the dynamic neural network from single-scale to multi-scale; (2) semi-supervised consistency regularization of pseudo-labels is introduced to model the inter-scale correlation of unannotated tissue types into a single end-to-end learning paradigm; and (3) superior scale-aware generalization is evidenced by directly applying a model trained on human kidney images to mouse kidney images, without retraining. By learning from ~150,000 human pathological image patches from six tissue types at three different resolutions, our approach achieved superior segmentation performance according to human visual assessment and evaluation of image-omics (i.e., spatial transcriptomics). The official implementation is available at https://github.com/ddrrnn123/Omni-Seg.