# RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification

arXiv: https://arxiv.org/abs/2202.11781 · HTML: https://mlsanity.com/paper/2202.11781/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 95 references that are in the map.
- First posted on arXiv 23 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Can AI help in screening Viral and COVID-19 pneumonia?](https://mlsanity.com/paper/2003.13145/) (2020)
- [Spatiotemporal Residual Networks for Video Action Recognition](https://mlsanity.com/paper/1611.02155/) (2016)
- [In the Eye of the Beholder: Gaze and Actions in First Person Video](https://mlsanity.com/paper/2006.00626/) (2020)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development](https://mlsanity.com/paper/2009.07386/) (2020)
- [Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions](https://mlsanity.com/paper/1901.01874/) (2019)
- [Integrating Human Gaze into Attention for Egocentric Activity Recognition](https://mlsanity.com/paper/2011.03920/) (2020)
- [Object Referring in Videos with Language and Human Gaze](https://mlsanity.com/paper/1801.01582/) (2018)
- [A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms](https://mlsanity.com/paper/1708.01817/) (2017)
- [COVID-Net S: Towards computer-aided severity assessment via training and validation of deep neural networks for geographic extent and opacity extent scoring of chest X-rays for SARS-CoV-2 lung disease severity](https://mlsanity.com/paper/2005.12855/) (2020)
- [Vision Transformer for COVID-19 CXR Diagnosis using Chest X-ray Feature Corpus](https://mlsanity.com/paper/2103.07055/) (2021)
- [Attention-based Multi-scale Gated Recurrent Encoder with Novel Correlation Loss for COVID-19 Progression Prediction](https://mlsanity.com/paper/2107.08330/) (2021)

## Built on by (most cited first, 19 of 19)

- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification](https://mlsanity.com/paper/2305.18221/) (2023)
- [Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning](https://mlsanity.com/paper/2205.12466/) (2022)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation](https://mlsanity.com/paper/2302.04308/) (2023)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [Gaze-directed Vision GNN for Mitigating Shortcut Learning in Medical Image](https://mlsanity.com/paper/2406.14050/) (2024)
- [Doctor-in-the-Loop: An Explainable, Multi-View Deep Learning Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer](https://mlsanity.com/paper/2502.17503/) (2025)
- [Revisiting Computer-Aided Tuberculosis Diagnosis](https://mlsanity.com/paper/2307.02848/) (2023)
- [RadGazeGen: Radiomics and Gaze-guided Medical Image Generation using Diffusion Models](https://mlsanity.com/paper/2410.00307/) (2024)
- [TranSOP: Transformer-based Multimodal Classification for Stroke Treatment Outcome Prediction](https://mlsanity.com/paper/2301.10829/) (2023)
- [Learning to search for and detect objects in foveal images using deep learning](https://mlsanity.com/paper/2304.05741/) (2023)
- [When Eye-Tracking Meets Machine Learning: A Systematic Review on Applications in Medical Image Analysis](https://mlsanity.com/paper/2403.07834/) (2024)
- [Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs](https://mlsanity.com/paper/2603.06697/) (2026)
- [Self-Supervised Curricular Deep Learning for Chest X-Ray Image Classification](https://mlsanity.com/paper/2301.10687/) (2023)
- [GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs](https://mlsanity.com/paper/2508.09478/) (2025)
- [Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis](https://mlsanity.com/paper/2602.21657/) (2026)
- [CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays](https://mlsanity.com/paper/2606.21020/) (2026)
- [FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification](https://mlsanity.com/paper/2603.22939/) (2026)

## Abstract (arXiv)

In this work, we present RadioTransformer, a novel visual attention-driven transformer framework, that leverages radiologists' gaze patterns and models their visuo-cognitive behavior for disease diagnosis on chest radiographs. Domain experts, such as radiologists, rely on visual information for medical image interpretation. On the other hand, deep neural networks have demonstrated significant promise in similar tasks even where visual interpretation is challenging. Eye-gaze tracking has been used to capture the viewing behavior of domain experts, lending insights into the complexity of visual search. However, deep learning frameworks, even those that rely on attention mechanisms, do not leverage this rich domain information. RadioTransformer fills this critical gap by learning from radiologists' visual search patterns, encoded as 'human visual attention regions' in a cascaded global-focal transformer framework. The overall 'global' image characteristics and the more detailed 'local' features are captured by the proposed global and focal modules, respectively. We experimentally validate the efficacy of our student-teacher approach for 8 datasets involving different disease classification tasks where eye-gaze data is not available during the inference phase. Code: https://github.com/bmi-imaginelab/radiotransformer.