# RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignment

arXiv: https://arxiv.org/abs/2501.07525 · HTML: https://mlsanity.com/paper/2501.07525/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 13 Jan 2025.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification](https://mlsanity.com/paper/2211.11158/) (2022)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation](https://mlsanity.com/paper/1805.08298/) (2018)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models](https://mlsanity.com/paper/2302.07257/) (2023)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports](https://mlsanity.com/paper/2004.12274/) (2020)
- [Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors](https://mlsanity.com/paper/2210.06340/) (2022)
- [Weakly Supervised Lesion Localization With Probabilistic-CAM Pooling](https://mlsanity.com/paper/2005.14480/) (2020)
- [AI can evolve without labels: self-evolving vision transformer for chest X-ray diagnosis through knowledge distillation](https://mlsanity.com/paper/2202.06431/) (2022)

## Built on by (most cited first, 12 of 12)

- [ReEvalMed: Rethinking Medical Report Evaluation by Aligning Metrics with Real-World Clinical Judgment](https://mlsanity.com/paper/2510.00280/) (2025)
- [Anatomy-VLM: A Fine-grained Vision-Language Model for Medical Interpretation](https://mlsanity.com/paper/2511.08402/) (2025)
- [Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration](https://mlsanity.com/paper/2505.02848/) (2025)
- [Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare](https://mlsanity.com/paper/2505.08818/) (2025)
- [EMAD: Evidence-Centric Grounded Multimodal Diagnosis for Alzheimer's Disease](https://mlsanity.com/paper/2602.19178/) (2026)
- [MedScribe: Clinically Grounded CT Reporting through Agentic Workflows](https://mlsanity.com/paper/2605.01779/) (2026)
- [RadPRISM: Schema-stratified radiology-report supervision for concept-disentangled image representations and visual grounding](https://mlsanity.com/paper/2608.00147/) (2026)
- [BUSTR: Descriptor-Aware Vision-Language Learning for Breast Ultrasound Report Generation](https://mlsanity.com/paper/2511.20956/) (2025)
- [LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation](https://mlsanity.com/paper/2602.04617/) (2026)
- [CLEAR-HPV: Interpretable concept discovery for human-papillomavirus-associated morphology in whole-slide histology](https://mlsanity.com/paper/2602.05126/) (2026)
- [Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning](https://mlsanity.com/paper/2603.08921/) (2026)
- [KANEx: Translating Kolmogorov-Arnold Networks' Interpretability to Medical Explainability](https://mlsanity.com/paper/2607.24730/) (2026)

## Abstract (arXiv)

Automated chest radiographs interpretation requires both accurate disease classification and detailed radiology report generation, presenting a significant challenge in the clinical workflow. Current approaches either focus on classification accuracy at the expense of interpretability or generate detailed but potentially unreliable reports through image captioning techniques. In this study, we present RadAlign, a novel framework that combines the predictive accuracy of vision-language models (VLMs) with the reasoning capabilities of large language models (LLMs). Inspired by the radiologist's workflow, RadAlign first employs a specialized VLM to align visual features with key medical concepts, achieving superior disease classification with an average AUC of 0.885 across multiple diseases. These recognized medical conditions, represented as text-based concepts in the aligned visual-language space, are then used to prompt LLM-based report generation. Enhanced by a retrieval-augmented generation mechanism that grounds outputs in similar historical cases, RadAlign delivers superior report quality with a GREEN score of 0.678, outperforming state-of-the-art methods' 0.634. Our framework maintains strong clinical interpretability while reducing hallucinations, advancing automated medical imaging and report analysis through integrated predictive and generative AI. Code is available at https://github.com/difeigu/RadAlign.