# GazeSearch: Radiology Findings Search Benchmark

arXiv: https://arxiv.org/abs/2411.05780 · HTML: https://mlsanity.com/paper/2411.05780/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 89 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening](https://mlsanity.com/paper/1903.08297/) (2019)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development](https://mlsanity.com/paper/2009.07386/) (2020)
- [DeepGaze II: Reading fixations from deep features trained on object recognition](https://mlsanity.com/paper/1610.01563/) (2016)
- [Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.14310/) (2020)
- [Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2108.11510/) (2021)
- [Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention](https://mlsanity.com/paper/2303.15274/) (2023)
- [REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays](https://mlsanity.com/paper/2109.14187/) (2021)
- [Finding any Waldo: zero-shot invariant and efficient visual search](https://mlsanity.com/paper/1807.10587/) (2018)
- [AiR: Attention with Reasoning Capability](https://mlsanity.com/paper/2007.14419/) (2020)
- [VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2211.15103/) (2022)
- [Unifying Top-down and Bottom-up Scanpath Prediction Using Transformers](https://mlsanity.com/paper/2303.09383/) (2023)
- [Target-absent Human Attention](https://mlsanity.com/paper/2207.01166/) (2022)
- [MaskDiff: Modeling Mask Distribution with Diffusion Probabilistic Model for Few-Shot Instance Segmentation](https://mlsanity.com/paper/2303.05105/) (2023)
- [AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation](https://mlsanity.com/paper/2210.02578/) (2022)

## Built on by (most cited first, 1 of 1)

- [A World Model of Radiologist Reading for Medical Image Representation Learning](https://mlsanity.com/paper/2605.23992/) (2026)

## Abstract (arXiv)

Medical eye-tracking data is an important information source for understanding how radiologists visually interpret medical images. This information not only improves the accuracy of deep learning models for X-ray analysis but also their interpretability, enhancing transparency in decision-making. However, the current eye-tracking data is dispersed, unprocessed, and ambiguous, making it difficult to derive meaningful insights. Therefore, there is a need to create a new dataset with more focus and purposeful eyetracking data, improving its utility for diagnostic applications. In this work, we propose a refinement method inspired by the target-present visual search challenge: there is a specific finding and fixations are guided to locate it. After refining the existing eye-tracking datasets, we transform them into a curated visual search dataset, called GazeSearch, specifically for radiology findings, where each fixation sequence is purposefully aligned to the task of locating a particular finding. Subsequently, we introduce a scan path prediction baseline, called ChestSearch, specifically tailored to GazeSearch. Finally, we employ the newly introduced GazeSearch as a benchmark to evaluate the performance of current state-of-the-art methods, offering a comprehensive assessment for visual search in the medical imaging domain. Code is available at urlhttps://github.com/UARK-AICV/GazeSearch.