mlsanityData to 7 Oct 2026

Multi stain graph fusion for multimodal integration in pathology

arXiv 2204.12541 ↗First posted 26 Apr 2022eess.IV cs.CV cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

In pathology, tissue samples are assessed using multiple staining techniques to enhance contrast in unique histologic features. In this paper, we introduce a multimodal CNN-GNN based graph fusion approach that leverages complementary information from multiple non-registered histopathology images to predict pathologic scores. We demonstrate this approach in nonalcoholic steatohepatitis (NASH) by predicting CRN fibrosis stage and NAFLD Activity Score (NAS). Primary assessment of NASH typically requires liver biopsy evaluation on two histological stains: Trichrome (TC) and hematoxylin and eosin (H&E). Our multimodal approach learns to extract complementary information from TC and H&E graphs corresponding to each stain while simultaneously learning an optimal policy to combine this information. We report up to 20% improvement in predicting fibrosis stage and NAS component grades over single-stain modeling approaches, measured by computing linearly weighted Cohen's kappa between machine-derived vs. pathologist consensus scores. Broadly, this paper demonstrates the value of leveraging diverse pathology images for improved ML-powered histologic assessment.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 22 of its 52 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

22 of 52 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
  9. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  10. 2015Gated Graph Sequence Neural Networkscited by 792
  11. 2019Self-Attention Graph Poolingcited by 421
  12. 2019Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathologycited by 172
10 more
  1. 2019Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosiscited by 165
  2. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 128
  3. 2016Deep Learning for Identifying Metastatic Breast Cancercited by 93
  4. 2017Machine learning methods for histopathological image analysiscited by 79
  5. 2020Hierarchical Inter-Message Passing for Learning on Molecular Graphscited by 59
  6. 2021A Survey on Graph-Based Deep Learning for Computational Histopathologycited by 41
  7. 2020Representation Learning of Histopathology Images using Graph Neural Networkscited by 40
  8. 2014On the Consistency of Ordinal Regression Methodscited by 24
  9. 2019Histographs: Graphs in Histopathologycited by 20
  10. 2020Lymph Node Gross Tumor Volume Detection in Oncology Imaging via Relationship Learning Using Graph Neural Networkcited by 12

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 3; 2025: 2.

11 papers.

  1. 2023Towards a Visual-Language Foundation Model for Computational Pathologycited by 52
  2. 2023AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Imagescited by 18
  3. 2024Cutting Through the Noise: Boosting LLM Performance on Math Word Problemscited by 14
  4. 2024Graph Neural Networks in Histopathology: Emerging Trends and Future Directionscited by 13
  5. 2022Multi-Scale Relational Graph Convolutional Network for Multiple Instance Learning in Histopathology Imagescited by 6
  6. 2024SHAP-CAT: A interpretable multi-modal framework enhancing WSI classification via virtual staining and shapley-value-based multimodal fusioncited by 3
  7. 2023A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping modelscited by 1
  8. 2025BioX-CPath: Biologically-driven Explainable Diagnostics for Multistain IHC Computational Pathologycited by 1
  9. 2025Is Exchangeability better than I.I.D to handle Data Distribution Shifts while Pooling Data for Data-scarce Medical image segmentation?cited by 1
  10. 2023MUSTANG: Multi-Stain Self-Attention Graph Multiple Instance Learning Pipeline for Histopathology Whole Slide Imagescited by 0
  11. 2023Correlation-Distance Graph Learning for Treatment Response Prediction from rs-fMRIcited by 0