# A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models

arXiv: https://arxiv.org/abs/2308.01328 · HTML: https://mlsanity.com/paper/2308.01328/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 62 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning](https://mlsanity.com/paper/2206.02647/) (2022)
- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Deep neural network models for computational histopathology: A survey](https://mlsanity.com/paper/1912.12378/) (2019)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [Multi-scale Domain-adversarial Multiple-instance CNN for Cancer Subtype Classification with Unannotated Histopathological Images](https://mlsanity.com/paper/2001.01599/) (2020)
- [Knowledge distillation from multi-modal to mono-modal segmentation networks](https://mlsanity.com/paper/2106.09564/) (2021)
- [BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix](https://mlsanity.com/paper/2204.11425/) (2022)
- [Fine-Tuning and Training of DenseNet for Histopathology Image Representation Using TCGA Diagnostic Slides](https://mlsanity.com/paper/2101.07903/) (2021)
- [Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology](https://mlsanity.com/paper/2203.00585/) (2022)
- [Similar Image Search for Histopathology: SMILY](https://mlsanity.com/paper/1901.11112/) (2019)
- [Self-supervised driven consistency training for annotation efficient histopathology image analysis](https://mlsanity.com/paper/2102.03897/) (2021)
- [Deep-Learning for Classification of Colorectal Polyps on Whole-Slide Images](https://mlsanity.com/paper/1703.01550/) (2017)
- [Prototype Knowledge Distillation for Medical Segmentation with Missing Modality](https://mlsanity.com/paper/2303.09830/) (2023)
- [AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images](https://mlsanity.com/paper/2303.00865/) (2023)
- [Multi stain graph fusion for multimodal integration in pathology](https://mlsanity.com/paper/2204.12541/) (2022)
- [Automated Diagnosis of Lymphoma with Digital Pathology Images Using Deep Learning](https://mlsanity.com/paper/1811.02668/) (2018)
- [Case-based Similar Image Retrieval for Weakly Annotated Large Histopathological Images of Malignant Lymphoma Using Deep Metric Learning](https://mlsanity.com/paper/2107.03602/) (2021)

## Built on by (most cited first, 1 of 1)

- [Histopathology image embedding based on foundation models features aggregation for patient treatment response prediction](https://mlsanity.com/paper/2408.03954/) (2024)

## Abstract (arXiv)

Determining lymphoma subtypes is a crucial step for better patient treatment targeting to potentially increase their survival chances. In this context, the existing gold standard diagnosis method, which relies on gene expression technology, is highly expensive and time-consuming, making it less accessibility. Although alternative diagnosis methods based on IHC (immunohistochemistry) technologies exist (recommended by the WHO), they still suffer from similar limitations and are less accurate. Whole Slide Image (WSI) analysis using deep learning models has shown promising potential for cancer diagnosis, that could offer cost-effective and faster alternatives to existing methods. In this work, we propose a vision transformer-based framework for distinguishing DLBCL (Diffuse Large B-Cell Lymphoma) cancer subtypes from high-resolution WSIs. To this end, we introduce a multi-modal architecture to train a classifier model from various WSI modalities. We then leverage this model through a knowledge distillation process to efficiently guide the learning of a mono-modal classifier. Our experimental study conducted on a lymphoma dataset of 157 patients shows the promising performance of our mono-modal classification model, outperforming six recent state-of-the-art methods. In addition, the power-law curve, estimated on our experimental data, suggests that with more training data from a reasonable number of additional patients, our model could achieve competitive diagnosis accuracy with IHC technologies. Furthermore, the efficiency of our framework is confirmed through an additional experimental study on an external breast cancer dataset (BCI dataset).