# CellViT++: Energy-Efficient and Adaptive Cell Segmentation and Classification Using Foundation Models

arXiv: https://arxiv.org/abs/2501.05269 · HTML: https://mlsanity.com/paper/2501.05269/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2025.

## Built on by (most cited first, 21 of 21)

- [Predicting Neoadjuvant Chemotherapy Response in Triple-Negative Breast Cancer Using Pre-Treatment Histopathologic Images](https://mlsanity.com/paper/2505.14730/) (2025)
- [Learning the Language of Histopathology Images reveals Prognostic Subgroups in Invasive Lung Adenocarcinoma Patients](https://mlsanity.com/paper/2508.16742/) (2025)
- [MorphGen: Morphology-Guided Representation Learning for Robust Single-Domain Generalization in Histopathological Cancer Classification](https://mlsanity.com/paper/2509.00311/) (2025)
- [MUSE: Multi-Scale Dense Self-Distillation for Nucleus Detection and Classification](https://mlsanity.com/paper/2511.05170/) (2025)
- [NuNext: Reframing Nucleus Detection as Next-Point Detection](https://mlsanity.com/paper/2603.07098/) (2026)
- [Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathology](https://mlsanity.com/paper/2603.08328/) (2026)
- [Genome-Anchored Foundation Model Embeddings Improve Molecular Prediction from Histology Images](https://mlsanity.com/paper/2506.19681/) (2025)
- [Leveraging Pathology Foundation Models for Panoptic Segmentation of Melanoma in H&E Images](https://mlsanity.com/paper/2507.13974/) (2025)
- [NucFuseRank: Dataset Fusion and Performance Ranking for Nuclei Instance Segmentation](https://mlsanity.com/paper/2601.20104/) (2026)
- [DeNuC: Decoupling Nuclei Detection and Classification in Histopathology](https://mlsanity.com/paper/2603.04240/) (2026)
- [Atlas H&E-TME: Scalable AI-Based Tissue Profiling at Expert Pathologist-Level Accuracy](https://mlsanity.com/paper/2606.12346/) (2026)
- [Cracking the PUMA Challenge in 24 Hours with CellViT++ and nnU-Net](https://mlsanity.com/paper/2503.12269/) (2025)
- [Evaluating New AI Cell Foundation Models on Challenging Kidney Pathology Cases Unaddressed by Previous Foundation Models](https://mlsanity.com/paper/2510.01287/) (2025)
- [Inferring Clinically Relevant Molecular Subtypes of Pancreatic Cancer from Routine Histopathology Using Deep Learning](https://mlsanity.com/paper/2601.03410/) (2026)
- [LEMON: a foundation model for nuclear morphology in Computational Pathology](https://mlsanity.com/paper/2603.25802/) (2026)
- [VitaminP: cross-modal learning enables whole-cell segmentation from routine histology](https://mlsanity.com/paper/2604.23799/) (2026)
- [Symb-xMIL: Symbolic Explanations for Multiple Instance Learning in Digital Pathology](https://mlsanity.com/paper/2606.06224/) (2026)
- [CellDETR: A Detection-Guided Framework for Scalable Cell Representation Learning from Histopathology Images](https://mlsanity.com/paper/2606.29463/) (2026)
- [SAFViT: Spatial Attention Fusion Gating for Vision Transformer-Based Nucleus Segmentation and Classification](https://mlsanity.com/paper/2607.27835/) (2026)
- [Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures](https://mlsanity.com/paper/2607.28007/) (2026)
- [Lymphocyte Mimicry Correction via Region-Level Tissue Reasoning and Unbalanced Optimal Transport](https://mlsanity.com/paper/2608.17151/) (2026)

## Abstract (arXiv)

Digital Pathology is a cornerstone in the diagnosis and treatment of diseases. A key task in this field is the identification and segmentation of cells in hematoxylin and eosin-stained images. Existing methods for cell segmentation often require extensive annotated datasets for training and are limited to a predefined cell classification scheme. To overcome these limitations, we propose CellViT^scriptscriptstyle ++, a framework for generalized cell segmentation in digital pathology. CellViT^scriptscriptstyle ++ utilizes Vision Transformers with foundation models as encoders to compute deep cell features and segmentation masks simultaneously. To adapt to unseen cell types, we rely on a computationally efficient approach. It requires minimal data for training and leads to a drastically reduced carbon footprint. We demonstrate excellent performance on seven different datasets, covering a broad spectrum of cell types, organs, and clinical settings. The framework achieves remarkable zero-shot segmentation and data-efficient cell-type classification. Furthermore, we show that CellViT^scriptscriptstyle ++ can leverage immunofluorescence stainings to generate training datasets without the need for pathologist annotations. The automated dataset generation approach surpasses the performance of networks trained on manually labeled data, demonstrating its effectiveness in creating high-quality training datasets without expert annotations. To advance digital pathology, CellViT^scriptscriptstyle ++ is available as an open-source framework featuring a user-friendly, web-based interface for visualization and annotation. The code is available under https://github.com/TIO-IKIM/CellViT-plus-plus.