mlsanityData to 7 Oct 2026

ChEX: Interactive Localization and Region Description in Chest X-rays

arXiv 2404.15770 ↗First posted 24 Apr 2024cs.CV cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Report generation models offer fine-grained textual interpretations of medical images like chest X-rays, yet they often lack interactivity (i.e. the ability to steer the generation process through user queries) and localized interpretability (i.e. visually grounding their predictions), which we deem essential for future adoption in clinical practice. While there have been efforts to tackle these issues, they are either limited in their interactivity by not supporting textual queries or fail to also offer localized interpretability. Therefore, we propose a novel multitask architecture and training paradigm integrating textual prompts and bounding boxes for diverse aspects like anatomical regions and pathologies. We call this approach the Chest X-Ray Explainer (ChEX). Evaluations across a heterogeneous set of 9 chest X-ray tasks, including localized image interpretation and report generation, showcase its competitiveness with SOTA models while additional analysis demonstrates ChEX's interactive capabilities. Code: https://github.com/philip-mueller/chex

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 69 of its 91 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

69 of 91 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2016Densely Connected Convolutional Networkscited by 6,691
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  11. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  12. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
38 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,147
  2. 2016Deep Networks with Stochastic Depthcited by 973
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  5. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  6. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  7. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  8. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  9. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  10. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  11. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  12. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  13. 2018Albumentations: fast and flexible image augmentationscited by 558
  14. 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 459
  15. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  16. 2021Conditional DETR for Fast Training Convergencecited by 422
  17. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
  18. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  19. 2023Towards Generalist Biomedical AIcited by 286
  20. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  21. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  22. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  23. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 236
  24. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  25. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  26. 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
  27. 2022Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detectioncited by 145
  28. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  29. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 139
  30. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
  31. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  32. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  33. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 114
  34. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
  35. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 93
  36. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 92
  37. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  38. 2021Class-agnostic Object Detection with Multi-modal Transformercited by 85

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 8.

18 papers.

Sort:

By area:

  1. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  2. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 24
  3. 2025MedGround-R1: Advancing Medical Image Grounding via Spatial-Semantic Rewarded Group Relative Policy Optimizationcited by 23
  4. 2025Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Reviewcited by 9
  5. 2026A unified multi-task framework enables interpretable chest radiograph analysiscited by 5
  6. 2026A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretationcited by 4
  7. 2026TRACE: Temporal Radiology with Anatomical Change Explanation for Grounded X-ray Report Generationcited by 2
  8. 2024Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generationcited by 1
  9. 2025Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptionscited by 1
  10. 2025OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generationcited by 1
  11. 2025ART-ASyn: Anatomy-aware Realistic Texture-based Anomaly Synthesis Framework for Chest X-Rayscited by 1
  12. 2025Generalised Medical Phrase Groundingcited by 1
6 more
  1. 2026AnatomiX, an Anatomy-Aware Grounded Multimodal Large Language Model for Chest X-Ray Interpretationcited by 1
  2. 2026Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoningcited by 1
  3. 2025Anatomy-Aware Conditional Image-Text Retrievalcited by 0
  4. 2026Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotationcited by 0
  5. 2026LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-raycited by 0
  6. 2026GRCD: Grounded Region Change Detection for Multi-Finding Chest X-Ray Pairscited by 0