mlsanityData to 7 Oct 2026

Chest ImaGenome Dataset for Clinical Reasoning

arXiv 2108.00316 ↗First posted 31 Jul 2021cs.CV cs.AI cs.CLcited by 89

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite the progress in automatic detection of radiologic findings from chest X-ray (CXR) images in recent years, a quantitative evaluation of the explainability of these models is hampered by the lack of locally labeled datasets for different findings. With the exception of a few expert-labeled small-scale datasets for specific findings, such as pneumonia and pneumothorax, most of the CXR deep learning models to date are trained on global "weak" labels extracted from text reports, or trained via a joint image and unstructured text learning strategy. Inspired by the Visual Genome effort in the computer vision community, we constructed the first Chest ImaGenome dataset with a scene graph data structure to describe 242,072 images. Local annotations are automatically produced using a joint rule-based natural language processing (NLP) and atlas-based bounding box detection pipeline. Through a radiologist constructed CXR ontology, the annotations for each CXR are connected as an anatomy-centered scene graph, useful for image-level reasoning and multimodal fusion applications. Overall, we provide: i) 1,256 combinations of relation annotations between 29 CXR anatomical locations (objects with bounding box coordinates) and their attributes, structured as a scene graph per image, ii) over 670,000 localized comparison relations (for improved, worsened, or no change) between the anatomical locations across sequential exams, as well as ii) a manually annotated gold standard scene graph dataset from 500 unique patients.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Builds on 25 of its 55 references that are in the map. First posted on arXiv 31 Jul 2021.

Builds on

25 of 55 references.

References · 25

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  6. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  10. 2018Object Hallucination in Image Captioningcited by 661
  11. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  12. 2016Visual Dialogcited by 377
13 more
  1. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  2. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  3. 2018Graph R-CNN for Scene Graph Generationcited by 296
  4. 2017Scene Graph Generation from Objects, Phrases and Region Captionscited by 154
  5. 2020When Radiology Report Generation Meets Knowledge Graphcited by 144
  6. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  7. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  8. 2019Clinically Accurate Chest X-Ray Report Generationcited by 138
  9. 2016Analyzing the Behavior of Visual Question Answering Modelscited by 119
  10. 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 112
  11. 2018Learning to Summarize Radiology Findingscited by 54
  12. 2016Semantic Compositional Networks for Visual Captioningcited by 50
  13. 2021AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-raycited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 17; 2024: 19; 2025: 23; 2026: 25.

89 papers.

Sort:

By area:

  1. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  2. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  3. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
  4. 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
  5. 2023MAIRA-1: A specialised large multimodal model for radiology report generationcited by 72
  6. 2023EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Imagescited by 53
  7. 2023Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answeringcited by 41
  8. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 32
  9. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 28
  10. 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
  11. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  12. 2024ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generationcited by 24
38 more
  1. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 24
  2. 2023Controllable Chest X-Ray Report Generation from Longitudinal Representationscited by 23
  3. 2023WBCAtt: A White Blood Cell Dataset Annotated with Detailed Morphological Attributescited by 22
  4. 2023RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoningcited by 22
  5. 2023Complex Organ Mask Guided Radiology Report Generationcited by 22
  6. 2022Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rayscited by 21
  7. 2022CheXRelNet: An Anatomy-Aware Model for Tracking Longitudinal Relationships between Chest X-Rayscited by 19
  8. 2024Anatomical Structure-Guided Medical Vision-Language Pre-trainingcited by 18
  9. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 18
  10. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  11. 2024Multimodal Large Language Model driven Radiology Report Generation with Clinical Knowledge Enhancementcited by 16
  12. 2024M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretationcited by 16
  13. 2023A Systematic Review of Deep Learning-based Research on Radiology Report Generationcited by 14
  14. 2023RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extractioncited by 11
  15. 2024FactCheXcker: Mitigating Measurement Hallucinations in Chest X-ray Report Generation Modelscited by 11
  16. 2025CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understandingcited by 11
  17. 2023Anatomy-Driven Pathology Detection on Chest X-rayscited by 10
  18. 2024How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?cited by 9
  19. 2025MMXU: A Multi-Modal and Multi-X-ray Understanding Dataset for Disease Progressioncited by 9
  20. 2025AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretationcited by 8
  21. 2025Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretationcited by 8
  22. 2022Self adaptive global-local feature enhancement for radiology report generationcited by 7
  23. 2024Uncovering Knowledge Gaps in Radiology Report Generation Models through Knowledge Graphscited by 7
  24. 2025DDaTR: Dynamic Difference-aware Temporal Residual Network for Longitudinal Radiology Report Generationcited by 7
  25. 2023Region-based Contrastive Pretraining for Medical Image Retrieval with Anatomic Querycited by 6
  26. 2023Finding-Aware Anatomical Tokens for Chest X-Ray Automated Reportingcited by 6
  27. 2024Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLMcited by 6
  28. 2024Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representationcited by 5
  29. 2025Anatomical grounding pre-training for medical phrase groundingcited by 5
  30. 2025CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rayscited by 5
  31. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationalecited by 5
  32. 2025Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environmentscited by 5
  33. 2026MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Datacited by 5
  34. 2023Expert Uncertainty and Severity Aware Chest X-Ray Classification by Multi-Relationship Graph Learningcited by 4
  35. 2025Walking the Tightrope: Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuningcited by 4
  36. 2025Modeling Clinical Uncertainty in Radiology Reports: from Explicit Uncertainty Markers to Implicit Reasoning Pathwayscited by 4
  37. 2026A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretationcited by 4
  38. 2025Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reportscited by 3

50 most cited of 89 All 89 (JSON)