# Multimodal Generalized Zero Shot Learning for Gleason Grading using Self-Supervised Learning

arXiv: https://arxiv.org/abs/2111.07646 · HTML: https://mlsanity.com/paper/2111.07646/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 147 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Self-Path: Self-supervision for Classification of Pathology Images with Limited Annotations](https://mlsanity.com/paper/2008.05571/) (2020)
- [Multi-modal Cycle-consistent Generalized Zero-Shot Learning](https://mlsanity.com/paper/1808.00136/) (2018)
- [Efficient Active Learning for Image Classification and Segmentation using a Sample Selection and Conditional Generative Adversarial Network](https://mlsanity.com/paper/1806.05473/) (2018)
- [Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2003.13261/) (2020)
- [Automated Gleason Grading of Prostate Biopsies using Deep Learning](https://mlsanity.com/paper/1907.07980/) (2019)
- [Generative Dual Adversarial Network for Generalized Zero-shot Learning](https://mlsanity.com/paper/1811.04857/) (2018)
- [Generalized Zero-Shot Learning Via Over-Complete Distribution](https://mlsanity.com/paper/2004.00666/) (2020)
- [Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding](https://mlsanity.com/paper/1910.10825/) (2019)
- [Chest X-rays Classification: A Multi-Label and Fine-Grained Problem](https://mlsanity.com/paper/1807.07247/) (2018)
- [Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition](https://mlsanity.com/paper/2104.11057/) (2021)
- [Structure Preserving Stain Normalization of Histopathology Images Using Self-Supervised Semantic Guidance](https://mlsanity.com/paper/2008.02101/) (2020)
- [Interpretability-Driven Sample Selection Using Self Supervised Learning For Disease Classification And Segmentation](https://mlsanity.com/paper/2104.06087/) (2021)
- [Joint Segmentation and Uncertainty Visualization of Retinal Layers in Optical Coherence Tomography Images using Bayesian Deep Learning](https://mlsanity.com/paper/1809.04282/) (2018)
- [Terabyte-scale Deep Multiple Instance Learning for Classification and Localization in Pathology](https://mlsanity.com/paper/1805.06983/) (2018)
- [Pathological Retinal Region Segmentation From OCT Images Using Geometric Relation Based Augmentation](https://mlsanity.com/paper/2003.14119/) (2020)
- [Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation](https://mlsanity.com/paper/2103.00528/) (2021)
- [Deep multiscale convolutional feature learning for weakly supervised localization of chest pathologies in X-ray images](https://mlsanity.com/paper/1808.08280/) (2018)
- [Night Time Haze and Glow Removal using Deep Dilated Convolutional Network](https://mlsanity.com/paper/1902.00855/) (2019)
- [Progressive Generative Adversarial Networks for Medical Image Super resolution](https://mlsanity.com/paper/1902.02144/) (2019)
- [Adversarial Pulmonary Pathology Translation for Pairwise Chest X-ray Data Augmentation](https://mlsanity.com/paper/1910.04961/) (2019)
- [Registration of Histopathogy Images Using Structural Information From Fine Grained Feature Maps](https://mlsanity.com/paper/2007.02078/) (2020)
- [Informative sample generation using class aware generative adversarial networks for classification of chest Xrays](https://mlsanity.com/paper/1904.10781/) (2019)
- [Retinal Vasculature Segmentation Using Local Saliency Maps and Generative Adversarial Networks For Image Super Resolution](https://mlsanity.com/paper/1710.04783/) (2017)
- [GAN Based Medical Image Registration](https://mlsanity.com/paper/1805.02369/) (2018)

## Built on by (most cited first, 4 of 4)

- [Unsupervised Domain Adaptation Using Feature Disentanglement And GCNs For Medical Image Classification](https://mlsanity.com/paper/2206.13123/) (2022)
- [Interpretable Saliency Maps And Self-Supervised Learning For Generalized Zero Shot Medical Image Classification](https://mlsanity.com/paper/2204.01728/) (2022)
- [Improved Super Resolution of MR Images Using CNNs and Vision Transformers](https://mlsanity.com/paper/2207.11748/) (2022)
- [Cyclic Generative Adversarial Networks With Congruent Image-Report Generation For Explainable Medical Image Analysis](https://mlsanity.com/paper/2211.08424/) (2022)

## Abstract (arXiv)

Gleason grading from histopathology images is essential for accurate prostate cancer (PCa) diagnosis. Since such images are obtained after invasive tissue resection quick diagnosis is challenging under the existing paradigm. We propose a method to predict Gleason grades from magnetic resonance (MR) images which are non-interventional and easily acquired. We solve the problem in a generalized zero-shot learning (GZSL) setting since we may not access training images of every disease grade. Synthetic MRI feature vectors of unseen grades (classes) are generated by exploiting Gleason grades' ordered nature through a conditional variational autoencoder (CVAE) incorporating self-supervised learning. Corresponding histopathology features are generated using cycle GANs, and combined with MR features to predict Gleason grades of test images. Experimental results show our method outperforms competing feature generating approaches for GZSL, and comes close to performance of fully supervised methods.