# GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification

arXiv: https://arxiv.org/abs/2305.18221 · HTML: https://mlsanity.com/paper/2305.18221/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Vision GNN: An Image is Worth Graph of Nodes](https://mlsanity.com/paper/2206.00272/) (2022)
- [Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation](https://mlsanity.com/paper/1603.08486/) (2016)
- [Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification](https://mlsanity.com/paper/1803.02315/) (2018)
- [Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development](https://mlsanity.com/paper/2009.07386/) (2020)
- [Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis](https://mlsanity.com/paper/2204.02976/) (2022)
- [Learning to recognize Abnormalities in Chest X-Rays with Location-Aware Dense Networks](https://mlsanity.com/paper/1803.04565/) (2018)
- [Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment](https://mlsanity.com/paper/2202.02838/) (2022)
- [Human Attention in Fine-grained Classification](https://mlsanity.com/paper/2111.01628/) (2021)
- [Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning](https://mlsanity.com/paper/2205.12466/) (2022)
- [RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification](https://mlsanity.com/paper/2202.11781/) (2022)
- [GazeSAM: What You See is What You Segment](https://mlsanity.com/paper/2304.13844/) (2023)
- [Abnormality Detection and Localization in Chest X-Rays using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1705.09850/) (2017)
- [A Collaborative Computer Aided Diagnosis (C-CAD) System with Eye-Tracking, Sparse Attentional Model, and Deep Learning](https://mlsanity.com/paper/1802.06260/) (2018)
- [Rectify ViT Shortcut Learning by Visual Saliency](https://mlsanity.com/paper/2206.08567/) (2022)
- [Gaze2Segment: A Pilot Study for Integrating Eye-Tracking Technology into Medical Image Segmentation](https://mlsanity.com/paper/1608.03235/) (2016)
- [Probabilistic Integration of Object Level Annotations in Chest X-ray Classification](https://mlsanity.com/paper/2210.06980/) (2022)
- [Gaze-Guided Class Activation Mapping: Leveraging Human Attention for Network Attention in Chest X-rays Classification](https://mlsanity.com/paper/2202.07107/) (2022)
- [Improving Disease Classification Performance and Explainability of Deep Learning Models in Radiology with Heatmap Generators](https://mlsanity.com/paper/2207.00157/) (2022)

## Built on by (most cited first, 25 of 25)

- [Improving Medical Multi-modal Contrastive Learning with Expert Annotations](https://mlsanity.com/paper/2403.10153/) (2024)
- [Trends, Applications, and Challenges in Human Attention Modelling](https://mlsanity.com/paper/2402.18673/) (2024)
- [Gaze-directed Vision GNN for Mitigating Shortcut Learning in Medical Image](https://mlsanity.com/paper/2406.14050/) (2024)
- [Grains of Saliency: Optimizing Saliency-based Training of Biometric Attack Detection Models](https://mlsanity.com/paper/2405.00650/) (2024)
- [Gaze-DETR: Using Expert Gaze to Reduce False Positives in Vulvovaginal Candidiasis Screening](https://mlsanity.com/paper/2405.09463/) (2024)
- [Enjoying Information Dividend: Gaze Track-based Medical Weakly Supervised Segmentation](https://mlsanity.com/paper/2505.22230/) (2025)
- [Recent Advances in Medical Image Classification](https://mlsanity.com/paper/2506.04129/) (2025)
- [Segment Anything for Video: A Comprehensive Review of Video Object Segmentation and Tracking from Past to Future](https://mlsanity.com/paper/2507.22792/) (2025)
- [Eyes on Target: Gaze-Aware Object Detection in Egocentric Video](https://mlsanity.com/paper/2511.01237/) (2025)
- [Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs](https://mlsanity.com/paper/2603.06697/) (2026)
- [Vision-language models for decoding provider attention during neonatal resuscitation](https://mlsanity.com/paper/2404.01207/) (2024)
- [Eyes Tell the Truth: GazeVal Highlights Shortcomings of Generative AI in Medical Imaging](https://mlsanity.com/paper/2503.20967/) (2025)
- [Saliency-Guided Training for Fingerprint Presentation Attack Detection](https://mlsanity.com/paper/2505.02176/) (2025)
- [GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs](https://mlsanity.com/paper/2508.09478/) (2025)
- [Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation](https://mlsanity.com/paper/2508.13068/) (2025)
- [Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis](https://mlsanity.com/paper/2602.21657/) (2026)
- [TransGUNet: Transformer Meets Graph-based Skip Connection for Medical Image Segmentation](https://mlsanity.com/paper/2502.09931/) (2025)
- [A Comparison of Object Detection and Phrase Grounding Models in Chest X-ray Abnormality Localization using Eye-tracking Data](https://mlsanity.com/paper/2503.01037/) (2025)
- [RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze](https://mlsanity.com/paper/2507.09097/) (2025)
- [From Attribution to Action: Jointly ALIGNing Predictions and Explanations](https://mlsanity.com/paper/2511.06944/) (2025)
- [SAGE: Saliency-Guided Contrastive Embeddings](https://mlsanity.com/paper/2511.12744/) (2025)
- [Structural Graph Neural Networks with Anatomical Priors for Explainable Chest X-ray Diagnosis](https://mlsanity.com/paper/2601.11987/) (2026)
- [GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans](https://mlsanity.com/paper/2603.00162/) (2026)
- [FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification](https://mlsanity.com/paper/2603.22939/) (2026)
- [Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention](https://mlsanity.com/paper/2607.23917/) (2026)

## Abstract (arXiv)

Eye tracking research is important in computer vision because it can help us understand how humans interact with the visual world. Specifically for high-risk applications, such as in medical imaging, eye tracking can help us to comprehend how radiologists and other medical professionals search, analyze, and interpret images for diagnostic and clinical purposes. Hence, the application of eye tracking techniques in disease classification has become increasingly popular in recent years. Contemporary works usually transform gaze information collected by eye tracking devices into visual attention maps (VAMs) to supervise the learning process. However, this is a time-consuming preprocessing step, which stops us from applying eye tracking to radiologists' daily work. To solve this problem, we propose a novel gaze-guided graph neural network (GNN), GazeGNN, to leverage raw eye-gaze data without being converted into VAMs. In GazeGNN, to directly integrate eye gaze into image classification, we create a unified representation graph that models both images and gaze pattern information. With this benefit, we develop a real-time, real-world, end-to-end disease classification algorithm for the first time in the literature. This achievement demonstrates the practicality and feasibility of integrating real-time eye tracking techniques into the daily work of radiologists. To our best knowledge, GazeGNN is the first work that adopts GNN to integrate image and eye-gaze data. Our experiments on the public chest X-ray dataset show that our proposed method exhibits the best classification performance compared to existing methods. The code is available at https://github.com/ukaukaaaa/GazeGNN.