# Leverage Weakly Annotation to Pixel-wise Annotation via Zero-shot Segment Anything Model for Molecular-empowered Learning

arXiv: https://arxiv.org/abs/2308.05785 · HTML: https://mlsanity.com/paper/2308.05785/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 10 of 10)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging](https://mlsanity.com/paper/2304.04155/) (2023)
- [All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning](https://mlsanity.com/paper/2307.00290/) (2023)
- [BoxCars: Improving Fine-Grained Recognition of Vehicles using 3-D Bounding Boxes in Traffic Surveillance](https://mlsanity.com/paper/1703.00686/) (2017)
- [Zero-shot Learning via Simultaneous Generating and Learning](https://mlsanity.com/paper/1910.09446/) (2019)
- [Weak Supervision for Generating Pixel-Level Annotations in Scene Text Segmentation](https://mlsanity.com/paper/1911.09026/) (2019)
- [Democratizing Pathological Image Segmentation with Lay Annotators via Molecular-empowered Learning](https://mlsanity.com/paper/2306.00047/) (2023)

## Built on by (most cited first, 4 of 4)

- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [Foundational Models for Pathology and Endoscopy Images: Application for Gastric Inflammation](https://mlsanity.com/paper/2406.18249/) (2024)
- [GeoVision Labeler: Zero-Shot Geospatial Classification with Vision and Language Models](https://mlsanity.com/paper/2505.24340/) (2025)
- [CoFi: A Fast Coarse-to-Fine Few-Shot Pipeline for Glomerular Basement Membrane Segmentation](https://mlsanity.com/paper/2508.11469/) (2025)

## Abstract (arXiv)

Precise identification of multiple cell classes in high-resolution Giga-pixel whole slide imaging (WSI) is critical for various clinical scenarios. Building an AI model for this purpose typically requires pixel-level annotations, which are often unscalable and must be done by skilled domain experts (e.g., pathologists). However, these annotations can be prone to errors, especially when distinguishing between intricate cell types (e.g., podocytes and mesangial cells) using only visual inspection. Interestingly, a recent study showed that lay annotators, when using extra immunofluorescence (IF) images for reference (referred to as molecular-empowered learning), can sometimes outperform domain experts in labeling. Despite this, the resource-intensive task of manual delineation remains a necessity during the annotation process. In this paper, we explore the potential of bypassing pixel-level delineation by employing the recent segment anything model (SAM) on weak box annotation in a zero-shot learning approach. Specifically, we harness SAM's ability to produce pixel-level annotations from box annotations and utilize these SAM-generated labels to train a segmentation model. Our findings show that the proposed SAM-assisted molecular-empowered learning (SAM-L) can diminish the labeling efforts for lay annotators by only requiring weak box annotations. This is achieved without compromising annotation accuracy or the performance of the deep learning-based segmentation. This research represents a significant advancement in democratizing the annotation process for training pathological image segmentation, relying solely on non-expert annotators.