mlsanityData to 14 Oct 2026

CellViT: Vision Transformers for Precise Cell Segmentation and Classification

arXiv 2306.15350 ↗First posted 27 Jun 2023eess.IV cs.CV cs.LGcited by 108

Not on the map. Why?

Abstract (arXiv)

Nuclei detection and segmentation in hematoxylin and eosin-stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer-based networks in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated Nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in-domain and out-of-domain pre-trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre-trained on 104 million histological image patches - achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F1-detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Builds on 38 of its 76 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

38 of 76 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Segment Anythingcited by 8,234
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017Mask R-CNNcited by 7,277
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
26 more
  1. 2015Fast R-CNNcited by 2,933
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  5. 2021UNETR: Transformers for 3D Medical Image Segmentationcited by 959
  6. 2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 830
  7. 2018Panoptic Segmentationcited by 802
  8. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
  10. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  11. 2018Albumentations: fast and flexible image augmentationscited by 560
  12. 2018HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Imagescited by 388
  13. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 359
  14. 2018Cell Detection with Star-convex Polygonscited by 183
  15. 2018A Novel Focal Tversky loss function with improved Attention U-Net for lesion segmentationcited by 172
  16. 2020U-Net and its variants for medical image segmentation: theory and applicationscited by 169
  17. 2020PanNuke Dataset Extension, Insights and Baselinescited by 114
  18. 2016DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentationcited by 84
  19. 2021Medical Image Segmentation Using Squeeze-and-Expansion Transformerscited by 57
  20. 2018Micro-Net: A unified model for segmentation of various objects in microscopy imagescited by 56
  21. 2021CoNIC: Colon Nuclei Identification and Counting Challenge 2022cited by 45
  22. 2022One Model is All You Need: Multi-Task Learning Enables Simultaneous Histology Image Segmentation and Classificationcited by 38
  23. 2021CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentationcited by 32
  24. 2022Nuclei instance segmentation and classification in histopathology images with StarDistcited by 31
  25. 2019Nuclear Instance Segmentation using a Proposal-Free Spatially Aware Deep Learning Frameworkcited by 10
  26. 2022Valuing Vicinity: Memory attention framework for context-based semantic segmentation in histopathologycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 22; 2025: 46; 2026: 36.

108 papers.

Sort:

By area:

  1. 2024BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at oncecited by 111
  2. 2024HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysiscited by 107
  3. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  4. 2024How to build the best medical image segmentation algorithm using foundation models: a comprehensive empirical study with Segment Anything Modelcited by 32
  5. 2024Foundation Models for Biomedical Image Segmentation: A Surveycited by 28
  6. 2024Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentationcited by 24
  7. 2023A Recent Survey of Vision Transformers for Medical Image Segmentationcited by 23
  8. 2024STimage-1K4M: A histopathology image-gene expression dataset for spatial transcriptomicscited by 23
  9. 2023Unleashing the Power of Prompt-driven Nucleus Instance Segmentationcited by 19
  10. 2024ViTGaze: Gaze Following with Interaction Features in Vision Transformerscited by 15
  11. 2025PixCell: A generative foundation model for digital histopathology imagescited by 11
  12. 2025Segment Anything for Histopathologycited by 9
38 more
  1. 2025PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditionscited by 9
  2. 2024CoTCoNet: An Optimized Coupled Transformer-Convolutional Network with an Adaptive Graph Reconstruction for Leukemia Detectioncited by 7
  3. 2024CellSeg1: Robust Cell Segmentation with One Training Imagecited by 6
  4. 2025MedicoSAM: Robust Improvement of SAM for Medical Imagingcited by 6
  5. 2024NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentationcited by 5
  6. 2024Evaluating Cell AI Foundation Models in Kidney Pathology with Human-in-the-Loop Enrichmentcited by 5
  7. 2025Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attentioncited by 5
  8. 2025Towards Effective and Efficient Context-aware Nucleus Detection in Histopathology Whole Slide Imagescited by 5
  9. 2025MambaVesselNet++: A Hybrid CNN-Mamba Architecture for Medical Image Segmentationcited by 5
  10. 2024OMG-Net: A Deep Learning Framework Deploying Segment Anything to Detect Pan-Cancer Mitotic Figures from Haematoxylin and Eosin-Stained Slidescited by 4
  11. 2024RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistencycited by 4
  12. 2025Revisiting Automatic Data Curation for Vision Foundation Models in Digital Pathologycited by 4
  13. 2025A Multi-Stage Auto-Context Deep Learning Framework for Tissue and Nuclei Segmentation and Classification in H&E-Stained Histological Images of Advanced Melanomacited by 4
  14. 2025Ensemble learning of pathology foundation models for precision oncologycited by 4
  15. 2025ACS-SegNet: An Attention-Based CNN-SegFormer Segmentation Network for Tissue Segmentation in Histopathologycited by 4
  16. 2023Morphological Profiling for Drug Discovery in the Era of Deep Learningcited by 3
  17. 2024LKCell: Efficient Cell Nuclei Instance Segmentation with Large Convolution Kernelscited by 3
  18. 2024Segmentation of Muscularis Propria in Colon Histopathology Images Using Vision Transformers for Hirschsprung's Diseasecited by 3
  19. 2025COIN: Confidence Score-Guided Distillation for Annotation-Free Cell Segmentationcited by 3
  20. 2025The Four Color Theorem for Cell Instance Segmentationcited by 3
  21. 2025Co-Seg++: Mutual Prompt-Guided Collaborative Learning for Versatile Medical Segmentationcited by 3
  22. 2025Co-Seg: Mutual Prompt-Guided Collaborative Learning for Tissue and Nuclei Segmentationcited by 3
  23. 2025MATCH: Multi-faceted Adaptive Topo-Consistency for Semi-Supervised Histopathology Segmentationcited by 3
  24. 2025Cell Instance Segmentation: The Devil Is in the Boundariescited by 3
  25. 2024NuLite -- Lightweight and Fast Model for Nuclei Instance Segmentation and Classificationcited by 2
  26. 2024Automated Classification of Cell Shapes: A Comparative Evaluation of Shape Descriptorscited by 2
  27. 2025Histo-Miner: Deep learning based tissue features extraction pipeline from H&E whole slide images of cutaneous squamous cell carcinomacited by 2
  28. 2025MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Modelscited by 2
  29. 2025Fine-grained Multi-class Nuclei Segmentation with Molecular-empowered All-in-SAM Modelcited by 2
  30. 2025EM3M: An Electron Micrograph Dataset for Microstructural Segmentation and Generationcited by 2
  31. 2025Learning the Language of Histopathology Images reveals Prognostic Subgroups in Invasive Lung Adenocarcinoma Patientscited by 2
  32. 2025MorphGen: Morphology-Guided Representation Learning for Robust Single-Domain Generalization in Histopathological Cancer Classificationcited by 2
  33. 2025PEaRL: Pathway-Enhanced Representation Learning for Gene and Pathway Expression Prediction from Histologycited by 2
  34. 2025MUSE: Multi-Scale Dense Self-Distillation for Nucleus Detection and Classificationcited by 2
  35. 2025Supervise Less, See More: Training-free Nuclear Instance Segmentation with Prototype-Guided Promptingcited by 2
  36. 2026Disco: Densely-overlapping Cell Instance Segmentation via Adjacency-aware Collaborative Coloringcited by 2
  37. 2026NuNext: Reframing Nucleus Detection as Next-Point Detectioncited by 2
  38. 2026Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathologycited by 2

50 most cited of 108 All 108 (JSON)