# IHC Matters: Incorporating IHC analysis to H&E Whole Slide Image Analysis for Improved Cancer Grading via Two-stage Multimodal Bilinear Pooling Fusion

arXiv: https://arxiv.org/abs/2405.08197 · HTML: https://mlsanity.com/paper/2405.08197/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 10 of 10)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix](https://mlsanity.com/paper/2204.11425/) (2022)
- [Deep Learning for Whole Slide Image Analysis: An Overview](https://mlsanity.com/paper/1910.11097/) (2019)
- [Deep Learning-enabled Virtual Histological Staining of Biological Samples](https://mlsanity.com/paper/2211.06822/) (2022)
- [Predicting Ki67, ER, PR, and HER2 Statuses from H&E-stained Breast Cancer Images](https://mlsanity.com/paper/2308.01982/) (2023)
- [The ACROBAT 2022 Challenge: Automatic Registration Of Breast Cancer Tissue](https://mlsanity.com/paper/2305.18033/) (2023)
- [convoHER2: A Deep Neural Network for Multi-Stage Classification of HER2 Breast Cancer](https://mlsanity.com/paper/2211.10690/) (2022)

## Built on by (most cited first, 2 of 2)

- [SHAP-CAT: A interpretable multi-modal framework enhancing WSI classification via virtual staining and shapley-value-based multimodal fusion](https://mlsanity.com/paper/2410.01408/) (2024)
- [Backdoor Learning in Language Models and Vision-Language Models](https://mlsanity.com/paper/2608.18095/) (2026)

## Abstract (arXiv)

Immunohistochemistry (IHC) plays a crucial role in pathology as it detects the over-expression of protein in tissue samples. However, there are still fewer machine learning model studies on IHC's impact on accurate cancer grading. We discovered that IHC and H&E possess distinct advantages and disadvantages while possessing certain complementary qualities. Building on this observation, we developed a two-stage multi-modal bilinear model with a feature pooling module. This model aims to maximize the potential of both IHC and HE's feature representation, resulting in improved performance compared to their individual use. Our experiments demonstrate that incorporating IHC data into machine learning models, alongside H&E stained images, leads to superior predictive results for cancer grading. The proposed framework achieves an impressive ACC higher of 0.953 on the public dataset BCI.