# Towards a text-based quantitative and explainable histopathology image analysis

arXiv: https://arxiv.org/abs/2407.07360 · HTML: https://mlsanity.com/paper/2407.07360/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Quilt-1M: One Million Image-Text Pairs for Histopathology](https://mlsanity.com/paper/2306.11207/) (2023)
- [BACH: Grand Challenge on Breast Cancer Histology Images](https://mlsanity.com/paper/1808.04277/) (2018)
- [Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation](https://mlsanity.com/paper/2203.06386/) (2022)
- [WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinoma](https://mlsanity.com/paper/2204.06455/) (2022)
- [Towards Inadequately Pre-trained Models in Transfer Learning](https://mlsanity.com/paper/2203.04668/) (2022)

## Built on by (most cited first, 7 of 7)

- [2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification](https://mlsanity.com/paper/2412.00678/) (2024)
- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [VLEER: Vision and Language Embeddings for Explainable Whole Slide Image Representation](https://mlsanity.com/paper/2502.20850/) (2025)
- [CLIP-IT: CLIP-based Pairing for Histology Images Classification](https://mlsanity.com/paper/2504.16181/) (2025)
- [Hierarchical Classification for Improved Histopathology Image Analysis](https://mlsanity.com/paper/2603.00504/) (2026)
- [Beyond H&E: Unlocking Pathological Insights with Polarization Imaging](https://mlsanity.com/paper/2503.05933/) (2025)
- [Normal and Abnormal Pathology Knowledge-Augmented Vision-Language Model for Anomaly Detection in Pathology Images](https://mlsanity.com/paper/2508.15256/) (2025)

## Abstract (arXiv)

Recently, vision-language pre-trained models have emerged in computational pathology. Previous works generally focused on the alignment of image-text pairs via the contrastive pre-training paradigm. Such pre-trained models have been applied to pathology image classification in zero-shot learning or transfer learning fashion. Herein, we hypothesize that the pre-trained vision-language models can be utilized for quantitative histopathology image analysis through a simple image-to-text retrieval. To this end, we propose a Text-based Quantitative and Explainable histopathology image analysis, which we call TQx. Given a set of histopathology images, we adopt a pre-trained vision-language model to retrieve a word-of-interest pool. The retrieved words are then used to quantify the histopathology images and generate understandable feature embeddings due to the direct mapping to the text description. To evaluate the proposed method, the text-based embeddings of four histopathology image datasets are utilized to perform clustering and classification tasks. The results demonstrate that TQx is able to quantify and analyze histopathology images that are comparable to the prevalent visual models in computational pathology.