# Point Transformer with Federated Learning for Predicting Breast Cancer HER2 Status from Hematoxylin and Eosin-Stained Whole Slide Images

arXiv: https://arxiv.org/abs/2312.06454 · HTML: https://mlsanity.com/paper/2312.06454/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 28 of 28)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2106.00908/) (2021)
- [Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2011.08939/) (2020)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Seesaw Loss for Long-Tailed Instance Segmentation](https://mlsanity.com/paper/2008.10032/) (2020)
- [Federated Learning for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2306.05980/) (2023)
- [ABC: Auxiliary Balanced Classifier for Class-imbalanced Semi-supervised Learning](https://mlsanity.com/paper/2110.10368/) (2021)
- [HarmoFL: Harmonizing Local and Global Drifts in Federated Learning on Heterogeneous Medical Images](https://mlsanity.com/paper/2112.10775/) (2021)
- [Transformers in 3D Point Clouds: A Survey](https://mlsanity.com/paper/2205.07417/) (2022)
- [Rethinking Normalization Methods in Federated Learning](https://mlsanity.com/paper/2210.03277/) (2022)
- [Her2 Challenge Contest: A Detailed Assessment of Automated Her2 Scoring Algorithms in Whole Slide Images of Breast Cancer Tissues](https://mlsanity.com/paper/1705.08369/) (2017)
- [HVTSurv: Hierarchical Vision Transformer for Patient-Level Survival Prediction from Whole Slide Image](https://mlsanity.com/paper/2306.17373/) (2023)
- [HEROHE Challenge: assessing HER2 status in breast cancer without immunohistochemistry or in situ hybridization](https://mlsanity.com/paper/2111.04738/) (2021)

## Built on by (most cited first, 1 of 1)

- [Federated and Transfer Learning for Cancer Detection Based on Image Analysis](https://mlsanity.com/paper/2405.20126/) (2024)

## Abstract (arXiv)

Directly predicting human epidermal growth factor receptor 2 (HER2) status from widely available hematoxylin and eosin (HE)-stained whole slide images (WSIs) can reduce technical costs and expedite treatment selection. Accurately predicting HER2 requires large collections of multi-site WSIs. Federated learning enables collaborative training of these WSIs without gigabyte-size WSIs transportation and data privacy concerns. However, federated learning encounters challenges in addressing label imbalance in multi-site WSIs from the real world. Moreover, existing WSI classification methods cannot simultaneously exploit local context information and long-range dependencies in the site-end feature representation of federated learning. To address these issues, we present a point transformer with federated learning for multi-site HER2 status prediction from HE-stained WSIs. Our approach incorporates two novel designs. We propose a dynamic label distribution strategy and an auxiliary classifier, which helps to establish a well-initialized model and mitigate label distribution variations across sites. Additionally, we propose a farthest cosine sampling based on cosine distance. It can sample the most distinctive features and capture the long-range dependencies. Extensive experiments and analysis show that our method achieves state-of-the-art performance at four sites with a total of 2687 WSIs. Furthermore, we demonstrate that our model can generalize to two unseen sites with 229 WSIs.