# REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays

arXiv: https://arxiv.org/abs/2109.14187 · HTML: https://mlsanity.com/paper/2109.14187/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 50 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 10 of 10)

- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning](https://mlsanity.com/paper/1711.05225/) (2017)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations](https://mlsanity.com/paper/2012.15029/) (2020)
- [Creation and Validation of a Chest X-Ray Dataset with Eye-tracking and Report Dictation for AI Development](https://mlsanity.com/paper/2009.07386/) (2020)
- [Thoracic Disease Identification and Localization with Limited Supervision](https://mlsanity.com/paper/1711.06373/) (2017)
- [Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning](https://mlsanity.com/paper/1901.01703/) (2019)
- [A Collaborative Computer Aided Diagnosis (C-CAD) System with Eye-Tracking, Sparse Attentional Model, and Deep Learning](https://mlsanity.com/paper/1802.06260/) (2018)

## Built on by (most cited first, 30 of 34)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review](https://mlsanity.com/paper/2403.02469/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Improving Medical Multi-modal Contrastive Learning with Expert Annotations](https://mlsanity.com/paper/2403.10153/) (2024)
- [Probabilistic Integration of Object Level Annotations in Chest X-ray Classification](https://mlsanity.com/paper/2210.06980/) (2022)
- [I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses](https://mlsanity.com/paper/2309.13550/) (2023)
- [Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification](https://mlsanity.com/paper/2403.04024/) (2024)
- [Comparing radiologists' gaze and saliency maps generated by interpretability methods for chest x-rays](https://mlsanity.com/paper/2112.11716/) (2021)
- [MENTOR: Human Perception-Guided Pretraining for Increased Generalization](https://mlsanity.com/paper/2310.19545/) (2023)
- [RadGazeGen: Radiomics and Gaze-guided Medical Image Generation using Diffusion Models](https://mlsanity.com/paper/2410.00307/) (2024)
- [FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation](https://mlsanity.com/paper/2411.15413/) (2024)
- [Localization supervision of chest x-ray classifiers using label-specific eye-tracking annotation](https://mlsanity.com/paper/2207.09771/) (2022)
- [MDF-Net for abnormality detection by fusing X-rays with clinical data](https://mlsanity.com/paper/2302.13390/) (2023)
- [CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation](https://mlsanity.com/paper/2604.10410/) (2026)
- [Integrating Eye-Gaze Data into CXR DL Approaches: A Preliminary study](https://mlsanity.com/paper/2302.02940/) (2023)
- [When Eye-Tracking Meets Machine Learning: A Systematic Review on Applications in Medical Image Analysis](https://mlsanity.com/paper/2403.07834/) (2024)
- [DALL-M: Context-Aware Clinical Data Augmentation with LLMs](https://mlsanity.com/paper/2407.08227/) (2024)
- [Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs](https://mlsanity.com/paper/2603.06697/) (2026)
- [CheXthought: A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation](https://mlsanity.com/paper/2604.26288/) (2026)
- [Improving X-ray Diagnostics through Eye-Tracking and XR](https://mlsanity.com/paper/2203.01643/) (2022)
- [Discrimination of Radiologists Utilizing Eye-Tracking Technology and Machine Learning: A Case Study](https://mlsanity.com/paper/2308.02748/) (2023)
- [Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity](https://mlsanity.com/paper/2404.03854/) (2024)
- [GazeSearch: Radiology Findings Search Benchmark](https://mlsanity.com/paper/2411.05780/) (2024)
- [Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation](https://mlsanity.com/paper/2508.13068/) (2025)
- [RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding](https://mlsanity.com/paper/2607.22293/) (2026)
- [Decoding Radiologists' Intentions: A Novel System for Accurate Region Identification in Chest X-ray Image Analysis](https://mlsanity.com/paper/2404.18981/) (2024)
- [Artificially Generated Visual Scanpath Improves Multi-label Thoracic Disease Classification in Chest X-Ray Images](https://mlsanity.com/paper/2503.00657/) (2025)
- [A Comparison of Object Detection and Phrase Grounding Models in Chest X-ray Abnormality Localization using Eye-tracking Data](https://mlsanity.com/paper/2503.01037/) (2025)
- [Eye-Tracking as a Tool to Quantify the Effects of CAD Display on Radiologists' Interpretation of Chest Radiographs](https://mlsanity.com/paper/2510.20864/) (2025)
- [IMACT-CXR: An Interactive Multi-Agent Conversational Tutoring System for Chest X-Ray Interpretation](https://mlsanity.com/paper/2511.15825/) (2025)

## Abstract (arXiv)

Deep learning has shown recent success in classifying anomalies in chest x-rays, but datasets are still small compared to natural image datasets. Supervision of abnormality localization has been shown to improve trained models, partially compensating for dataset sizes. However, explicitly labeling these anomalies requires an expert and is very time-consuming. We propose a potentially scalable method for collecting implicit localization data using an eye tracker to capture gaze locations and a microphone to capture a dictation of a report, imitating the setup of a reading room. The resulting REFLACX (Reports and Eye-Tracking Data for Localization of Abnormalities in Chest X-rays) dataset was labeled across five radiologists and contains 3,032 synchronized sets of eye-tracking data and timestamped report transcriptions for 2,616 chest x-rays from the MIMIC-CXR dataset. We also provide auxiliary annotations, including bounding boxes around lungs and heart and validation labels consisting of ellipses localizing abnormalities and image-level labels. Furthermore, a small subset of the data contains readings from all radiologists, allowing for the calculation of inter-rater scores.