A unified multi-task framework enables interpretable chest radiograph analysis
Not on the map. Why?
Abstract (arXiv)
While multimodal deep learning has advanced medical imaging analysis, existing black-box systems textcolorblackmay remain confined to isolated tasks, often overlooking the trust-sensitive nature of clinical diagnosis as a multi-task process. We propose IMT-CXR (Interpretable Multi-task Transformer for Chest X-ray Analysis), a framework that emulates radiologists' diagnostic workflow through three evidence-driven stages: 1) Disease recognition; 2) Attribute characterization (e.g., size, location, severity quantification); 3) Evidence-integrated report generation with traceable decision pathways. The framework employs a unified transformer architecture optimized via medical-domain instruction tuning, sequentially executing four clinical tasks: multi-label disease classification, lesion localization, anatomical segmentation, and radiology report generation. Experimental validation demonstrates competitive performance on ten CXR benchmarks under direct inference and fine-tuning settings. In a blinded evaluation of 160 historical reports from four medical centers, three radiologists rated 66% of AI-generated reports as comparable to or surpassing original clinical reports in diagnostic clarity, highlighting the framework's translational potential. By establishing traceable diagnostic pathways from anatomical findings to conclusions, this work bridges the gap between AI technical metrics and clinical utility, advancing trustworthy AI systems in medical imaging.
Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 63 of its 82 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
63 of 82 references.
References · 63
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
- 2022Training language models to follow instructions with human feedbackcited by 15,856
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
- 2023Visual Instruction Tuningcited by 7,827
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
38 more
- 2022Large Language Models Encode Clinical Knowledgecited by 1,817
- 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,419
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,296
- 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 970
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 896
- 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
- 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 744
- 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 639
- 2016Self-critical Sequence Training for Image Captioningcited by 544
- 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
- 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
- 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 369
- 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
- 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 338
- 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
- 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 238
- 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
- 2022Cross-modal Memory Networks for Radiology Report Generationcited by 184
- 2022Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learningcited by 159
- 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
- 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
- 2024MAIRA-2: Grounded Radiology Report Generationcited by 142
- 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 140
- 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
- 2023Knowledge-enhanced Visual-Language Pre-training on Chest Radiology Imagescited by 128
- 2022Multimodal Masked Autoencoders Learn Transferable Representationscited by 109
- 2021Deep Learning for Chest X-ray Analysis: A Surveycited by 84
- 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80
- 2023Advancing Radiograph Representation Learning with Masked Record Modelingcited by 75
- 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
- 2022Delving into Masked Autoencoders for Multi-Label Thorax Disease Classificationcited by 65
- 2023ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoderscited by 50
- 2021Visual Grounding with Transformerscited by 42
- 2024MedVersa: A Generalist Foundation Model for Medical Image Interpretationcited by 38
- 2023Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generationcited by 37
- 2021A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentationcited by 33
- 2023CheXmask: a large-scale dataset of anatomical segmentation masks for multi-center chest x-ray imagescited by 33
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Stepwise Token Selection for Efficient Multimodal Large Language Modelscited by 5
- 2026Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoningcited by 4
- 2026PathSelect: Sequential Token Selection for Whole Slide Pathologycited by 2
- 2026DiffPrune: differentiable information throttling for token pruning in vision-language modelscited by 1
- 2026StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Modelscited by 0
- 2026Pre-training, Reasoning, Benchmarking: X-ray Report Generation on CheXpert Plus Datasetcited by 0
- 2026Pre-training, Reasoning, Benchmarking: X-ray Report Generation on CheXpert Plus Dataset23 Sep 2026
- 2026StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Models12 Sep 2026
- 2026DiffPrune: differentiable information throttling for token pruning in vision-language models3 Aug 2026
- 2026PathSelect: Sequential Token Selection for Whole Slide Pathology26 Jul 2026
- 2026Stepwise Token Selection for Efficient Multimodal Large Language Models14 Jun 2026
- 2026Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoning7 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2023I-AI: A Controllable & Interpretable AI System for Decoding Radiologists' Intense Focus for Accurate CXR Diagnoses
- 2025CXR-LanIC: Language-Grounded Interpretable Classifier for Chest X-Ray Diagnosis
- 2024M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation
- 2025Revolutionizing Radiology Workflow with Factual and Efficient CXR Report Generation
- 2024Towards Interpretable Radiology Report Generation via Concept Bottlenecks using a Multi-Agentic RAG
- 2025VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedback
- 2025A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography
- 2025X-Ray-CoT: Interpretable Chest X-ray Diagnosis with Vision-Language Models via Chain-of-Thought Reasoning