# FG-CXR: A Radiologist-Aligned Gaze Dataset for Enhancing Interpretability in Chest X-Ray Report Generation arXiv: https://arxiv.org/abs/2411.15413 · HTML: https://mlsanity.com/paper/2411.15413/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 54 references that are in the map. - First posted on arXiv 23 Nov 2024. ## Builds on (most cited first, 30 of 31) - [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016) - [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019) - [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019) - [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017) - [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017) - [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021) - [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018) - [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017) - [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021) - [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023) - [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020) - [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021) - [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019) - [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019) - [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022) - [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021) - [When Radiology Report Generation Meets Knowledge Graph](https://mlsanity.com/paper/2002.08277/) (2020) - [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018) - [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023) - [Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images](https://mlsanity.com/paper/1703.10660/) (2017) - [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022) - [AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation](https://mlsanity.com/paper/2203.10095/) (2022) - [Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports](https://mlsanity.com/paper/2004.12274/) (2020) - [Music-Driven Group Choreography](https://mlsanity.com/paper/2303.12337/) (2023) - [Move Forward and Tell: A Progressive Generator of Video Descriptions](https://mlsanity.com/paper/1807.10018/) (2018) - [REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays](https://mlsanity.com/paper/2109.14187/) (2021) - [EmbryosFormer: Deformable Transformer and Collaborative Encoding-Decoding for Embryos Stage Development Classification](https://mlsanity.com/paper/2210.04615/) (2022) - [I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses](https://mlsanity.com/paper/2309.13550/) (2023) - [∞-Brush: Controllable Large Image Synthesis with Diffusion Models in Infinite Dimensions](https://mlsanity.com/paper/2407.14709/) (2024) - [Looking in the Right place for Anomalies: Explainable AI through Automatic Location Learning](https://mlsanity.com/paper/2008.00363/) (2020) ## Built on by (most cited first, 4 of 4) - [Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation](https://mlsanity.com/paper/2509.24739/) (2025) - [Look & Mark: Leveraging Radiologist Eye Fixations and Bounding boxes in Multimodal Large Language Models for Chest X-ray Report Generation](https://mlsanity.com/paper/2505.22222/) (2025) - [Explainable Artificial Intelligence in Biomedical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2507.07148/) (2025) - [GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans](https://mlsanity.com/paper/2603.00162/) (2026) ## Abstract (arXiv) Developing an interpretable system for generating reports in chest X-ray (CXR) analysis is becoming increasingly crucial in Computer-aided Diagnosis (CAD) systems, enabling radiologists to comprehend the decisions made by these systems. Despite the growth of diverse datasets and methods focusing on report generation, there remains a notable gap in how closely these models' generated reports align with the interpretations of real radiologists. In this study, we tackle this challenge by initially introducing Fine-Grained CXR (FG-CXR) dataset, which provides fine-grained paired information between the captions generated by radiologists and the corresponding gaze attention heatmaps for each anatomy. Unlike existing datasets that include a raw sequence of gaze alongside a report, with significant misalignment between gaze location and report content, our FG-CXR dataset offers a more grained alignment between gaze attention and diagnosis transcript. Furthermore, our analysis reveals that simply applying black-box image captioning methods to generate reports cannot adequately explain which information in CXR is utilized and how long needs to attend to accurately generate reports. Consequently, we propose a novel explainable radiologist's attention generator network (Gen-XAI) that mimics the diagnosis process of radiologists, explicitly constraining its output to closely align with both radiologist's gaze attention and transcript. Finally, we perform extensive experiments to illustrate the effectiveness of our method. Our datasets and checkpoint is available at https://github.com/UARK-AICV/FG-CXR.