# One Model is All You Need: Multi-Task Learning Enables Simultaneous Histology Image Segmentation and Classification

arXiv: https://arxiv.org/abs/2203.00077 · HTML: https://mlsanity.com/paper/2203.00077/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 70 references that are in the map.
- First posted on arXiv 28 Feb 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Gland Segmentation in Colon Histology Images: The GlaS Challenge Contest](https://mlsanity.com/paper/1603.00275/) (2016)
- [Self supervised contrastive learning for digital histopathology](https://mlsanity.com/paper/2011.13971/) (2020)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [Lizard: A Large-Scale Dataset for Colonic Nuclear Instance Segmentation and Classification](https://mlsanity.com/paper/2108.11195/) (2021)
- [MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Images](https://mlsanity.com/paper/1806.01963/) (2018)
- [Automated Gleason Grading of Prostate Biopsies using Deep Learning](https://mlsanity.com/paper/1907.07980/) (2019)
- [Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles](https://mlsanity.com/paper/2103.05121/) (2021)
- [Trace Norm Regularised Deep Multi-Task Learning](https://mlsanity.com/paper/1606.04038/) (2016)
- [Many Task Learning with Task Routing](https://mlsanity.com/paper/1903.12117/) (2019)
- [CoNIC: Colon Nuclei Identification and Counting Challenge 2022](https://mlsanity.com/paper/2111.14485/) (2021)
- [Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Images](https://mlsanity.com/paper/1907.09478/) (2019)
- [Multi-task pre-training of deep neural networks for digital pathology](https://mlsanity.com/paper/2005.02561/) (2020)

## Built on by (most cited first, 30 of 38)

- [CellViT: Vision Transformers for Precise Cell Segmentation and Classification](https://mlsanity.com/paper/2306.15350/) (2023)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [Overcoming Data Scarcity in Biomedical Imaging with a Foundational Multi-Task Model](https://mlsanity.com/paper/2311.09847/) (2023)
- [Domain Generalization in Computational Pathology: Survey and Guidelines](https://mlsanity.com/paper/2310.19656/) (2023)
- [Evidence-based diagnostic reasoning with multi-agent copilot for human pathology](https://mlsanity.com/paper/2506.20964/) (2025)
- [A Foundational Multimodal Vision Language AI Assistant for Human Pathology](https://mlsanity.com/paper/2312.07814/) (2023)
- [Structure Embedded Nucleus Classification for Histopathology Images](https://mlsanity.com/paper/2302.11416/) (2023)
- [Deep Learning in Breast Cancer Imaging: A Decade of Progress and Future Directions](https://mlsanity.com/paper/2304.06662/) (2023)
- [Weakly Supervised Joint Whole-Slide Segmentation and Classification in Prostate Cancer](https://mlsanity.com/paper/2301.02933/) (2023)
- [Tissue Concepts: supervised foundation models in computational pathology](https://mlsanity.com/paper/2409.03519/) (2024)
- [Breast Cancer Immunohistochemical Image Generation: a Benchmark Dataset and Challenge Review](https://mlsanity.com/paper/2305.03546/) (2023)
- [MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis](https://mlsanity.com/paper/2308.16561/) (2023)
- [VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysis](https://mlsanity.com/paper/2410.08397/) (2024)
- [A Dual-branch Self-supervised Representation Learning Framework for Tumour Segmentation in Whole Slide Images](https://mlsanity.com/paper/2303.11019/) (2023)
- [Dataset Distillation for Histopathology Image Classification](https://mlsanity.com/paper/2408.09709/) (2024)
- [A Multi-Stage Auto-Context Deep Learning Framework for Tissue and Nuclei Segmentation and Classification in H&E-Stained Histological Images of Advanced Melanoma](https://mlsanity.com/paper/2503.23958/) (2025)
- [Valuing Vicinity: Memory attention framework for context-based semantic segmentation in histopathology](https://mlsanity.com/paper/2210.11822/) (2022)
- [A Fully Automated and Explainable Algorithm for the Prediction of Malignant Transformation in Oral Epithelial Dysplasia](https://mlsanity.com/paper/2307.03757/) (2023)
- [Benchmarking Domain Generalization Algorithms in Computational Pathology](https://mlsanity.com/paper/2409.17063/) (2024)
- [Co-Seg++: Mutual Prompt-Guided Collaborative Learning for Versatile Medical Segmentation](https://mlsanity.com/paper/2506.17159/) (2025)
- [Joint multi-task learning improves weakly-supervised biomarker prediction in computational pathology](https://mlsanity.com/paper/2403.03891/) (2024)
- [Resource Efficient Multi-stain Kidney Glomeruli Segmentation via Self-supervision](https://mlsanity.com/paper/2412.15389/) (2024)
- [ModalTune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-task Learning in Digital Pathology](https://mlsanity.com/paper/2503.17564/) (2025)
- [Recent Advances in Medical Image Classification](https://mlsanity.com/paper/2506.04129/) (2025)
- [Opportunities and challenges in the application of large artificial intelligence models in radiology](https://mlsanity.com/paper/2403.16112/) (2024)
- [Adapting Foundation Models for Few-Shot Medical Image Segmentation: Actively and Sequentially](https://mlsanity.com/paper/2502.01000/) (2025)
- [A three in one bottom-up framework for simultaneous semantic segmentation, instance segmentation and classification of multi-organ nuclei in digital cancer histology](https://mlsanity.com/paper/2308.11179/) (2023)
- [A Classification-Based Adaptive Segmentation Pipeline: Feasibility Study Using Polycystic Liver Disease and Metastases from Colorectal Cancer CT Images](https://mlsanity.com/paper/2405.01644/) (2024)
- [Selecting the Best Sequential Transfer Path for Medical Image Segmentation with Limited Labeled Data](https://mlsanity.com/paper/2410.06892/) (2024)
- [Contrastive Deep Learning Reveals Age Biomarkers in Histopathological Skin Biopsies](https://mlsanity.com/paper/2411.16956/) (2024)

## Abstract (arXiv)

The recent surge in performance for image analysis of digitised pathology slides can largely be attributed to the advances in deep learning. Deep models can be used to initially localise various structures in the tissue and hence facilitate the extraction of interpretable features for biomarker discovery. However, these models are typically trained for a single task and therefore scale poorly as we wish to adapt the model for an increasing number of different tasks. Also, supervised deep learning models are very data hungry and therefore rely on large amounts of training data to perform well. In this paper, we present a multi-task learning approach for segmentation and classification of nuclei, glands, lumina and different tissue regions that leverages data from multiple independent data sources. While ensuring that our tasks are aligned by the same tissue type and resolution, we enable meaningful simultaneous prediction with a single network. As a result of feature sharing, we also show that the learned representation can be used to improve the performance of additional tasks via transfer learning, including nuclear classification and signet ring cell detection. As part of this work, we train our developed Cerberus model on a huge amount of data, consisting of over 600K objects for segmentation and 440K patches for classification. We use our approach to process 599 colorectal whole-slide images from TCGA, where we localise 377 million, 900K and 2.1 million nuclei, glands and lumina, respectively and make the results available to the community for downstream analysis.