# Random Token Fusion for Multi-View Medical Diagnosis

arXiv: https://arxiv.org/abs/2410.15847 · HTML: https://mlsanity.com/paper/2410.15847/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [Is it Time to Replace CNNs with Transformers for Medical Images?](https://mlsanity.com/paper/2108.09038/) (2021)
- [TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers](https://mlsanity.com/paper/2207.08409/) (2022)
- [Multi-view analysis of unregistered medical images using cross-view transformers](https://mlsanity.com/paper/2103.11390/) (2021)
- [MVT: Multi-view Vision Transformer for 3D Object Recognition](https://mlsanity.com/paper/2110.13083/) (2021)
- [PatchDropout: Economizing Vision Transformers Using Patch Dropout](https://mlsanity.com/paper/2208.07220/) (2022)
- [TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers](https://mlsanity.com/paper/2210.07562/) (2022)
- [Act Like a Radiologist: Towards Reliable Multi-view Correspondence Reasoning for Mammogram Mass Detection](https://mlsanity.com/paper/2105.10160/) (2021)
- [Quantifying the Value of Lateral Views in Deep Learning for Chest X-rays](https://mlsanity.com/paper/2002.02582/) (2020)
- [A novel multi-view deep learning approach for BI-RADS and density assessment of mammograms](https://mlsanity.com/paper/2112.04490/) (2021)
- [MixPro: Data Augmentation with MaskMix and Progressive Attention Labeling for Vision Transformer](https://mlsanity.com/paper/2304.12043/) (2023)
- [PatchUp: A Feature-Space Block-Level Regularization Technique for Convolutional Neural Networks](https://mlsanity.com/paper/2006.07794/) (2020)
- [CheXFusion: Effective Fusion of Multi-View Features using Transformers for Long-Tailed Chest X-Ray Classification](https://mlsanity.com/paper/2308.03968/) (2023)
- [Transformers Improve Breast Cancer Diagnosis from Unregistered Multi-View Mammograms](https://mlsanity.com/paper/2206.10096/) (2022)
- [M&M: Tackling False Positives in Mammography with a Multi-view and Multi-instance Learning Sparse Detector](https://mlsanity.com/paper/2308.06420/) (2023)

## Abstract (arXiv)

In multi-view medical diagnosis, deep learning-based models often fuse information from different imaging perspectives to improve diagnostic performance. However, existing approaches are prone to overfitting and rely heavily on view-specific features, which can lead to trivial solutions. In this work, we introduce Random Token Fusion (RTF), a novel technique designed to enhance multi-view medical image analysis using vision transformers. By integrating randomness into the feature fusion process during training, RTF addresses the issue of overfitting and enhances the robustness and accuracy of diagnostic models without incurring any additional cost at inference. We validate our approach on standard mammography and chest X-ray benchmark datasets. Through extensive experiments, we demonstrate that RTF consistently improves the performance of existing fusion methods, paving the way for a new generation of multi-view medical foundation models.