# Improving mitosis detection on histopathology images using large vision-language models

arXiv: https://arxiv.org/abs/2310.07176 · HTML: https://mlsanity.com/paper/2310.07176/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations](https://mlsanity.com/paper/2008.05571/) (2020)
- [Mitosis domain generalization in histopathology images -- The MIDOG challenge](https://mlsanity.com/paper/2204.03742/) (2022)
- [Towards a Visual-Language Foundation Model for Computational Pathology](https://mlsanity.com/paper/2307.12914/) (2023)

## Built on by (most cited first, 3 of 3)

- [Medical Vision Language Pretraining: A survey](https://mlsanity.com/paper/2312.06224/) (2023)
- [Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainability](https://mlsanity.com/paper/2406.12897/) (2024)
- [MiTHras: Task-specific Hierarchical Semi-supervised Contrastive Masked Autoencoder for Mitotic Figure Analysis](https://mlsanity.com/paper/2609.24736/) (2026)

## Abstract (arXiv)

In certain types of cancerous tissue, mitotic count has been shown to be associated with tumor proliferation, poor prognosis, and therapeutic resistance. Due to the high inter-rater variability of mitotic counting by pathologists, convolutional neural networks (CNNs) have been employed to reduce the subjectivity of mitosis detection in hematoxylin and eosin (H&E)-stained whole slide images. However, most existing models have performance that lags behind expert panel review and only incorporate visual information. In this work, we demonstrate that pre-trained large-scale vision-language models that leverage both visual features and natural language improve mitosis detection accuracy. We formulate the mitosis detection task as an image captioning task and a visual question answering (VQA) task by including metadata such as tumor and scanner types as context. The effectiveness of our pipeline is demonstrated via comparison with various baseline models using 9,501 mitotic figures and 11,051 hard negatives (non-mitotic figures that are difficult to characterize) from the publicly available Mitosis Domain Generalization Challenge (MIDOG22) dataset.