# Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation

arXiv: https://arxiv.org/abs/2312.08078 · HTML: https://mlsanity.com/paper/2312.08078/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [From Captions to Visual Concepts and Back](https://mlsanity.com/paper/1411.4952/) (2014)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains](https://mlsanity.com/paper/2210.04133/) (2022)
- [Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment](https://mlsanity.com/paper/2112.15011/) (2021)
- [Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias](https://mlsanity.com/paper/2305.19894/) (2023)
- [DPT: Deformable Patch-based Transformer for Visual Recognition](https://mlsanity.com/paper/2107.14467/) (2021)
- [Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining](https://mlsanity.com/paper/2107.14572/) (2021)
- [Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment](https://mlsanity.com/paper/2008.09884/) (2020)
- [IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training](https://mlsanity.com/paper/2310.07355/) (2023)
- [Unifying Vision-Language Representation Space with Single-tower Transformer](https://mlsanity.com/paper/2211.11153/) (2022)
- [Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment](https://mlsanity.com/paper/2109.01949/) (2021)
- [LIMITR: Leveraging Local Information for Medical Image-Text Representation](https://mlsanity.com/paper/2303.11755/) (2023)
- [Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training](https://mlsanity.com/paper/2307.07246/) (2023)
- [Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining](https://mlsanity.com/paper/2304.14204/) (2023)
- [COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignment](https://mlsanity.com/paper/2308.03475/) (2023)

## Built on by (most cited first, 26 of 26)

- [Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation](https://mlsanity.com/paper/2502.20056/) (2025)
- [Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation](https://mlsanity.com/paper/2407.02235/) (2024)
- [A Survey on Trustworthiness in Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2407.15851/) (2024)
- [Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation](https://mlsanity.com/paper/2501.03458/) (2025)
- [MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.03320/) (2026)
- [Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuning](https://mlsanity.com/paper/2506.15477/) (2025)
- [DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching](https://mlsanity.com/paper/2412.14535/) (2024)
- [MCA-RG: Enhancing LLMs with Medical Concept Alignment for Radiology Report Generation](https://mlsanity.com/paper/2507.06992/) (2025)
- [EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation](https://mlsanity.com/paper/2510.16776/) (2025)
- [Generative AI for Healthcare: Fundamentals, Challenges, and Perspectives](https://mlsanity.com/paper/2510.24551/) (2025)
- [Anatomy-Guided Radiology Report Generation with Pathology-Aware Regional Prompts](https://mlsanity.com/paper/2411.10789/) (2024)
- [SigVLP: Sigmoid Volume-Language Pre-Training for Self-Supervised CT-Volume Adaptive Representation Learning](https://mlsanity.com/paper/2602.21735/) (2026)
- [Discrete Diffusion Models with MLLMs for Unified Medical Multimodal Generation](https://mlsanity.com/paper/2510.06131/) (2025)
- [FaNe: Towards Fine-Grained Cross-Modal Contrast with False-Negative Reduction and Text-Conditioned Sparse Attention](https://mlsanity.com/paper/2511.12215/) (2025)
- [RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation](https://mlsanity.com/paper/2604.27559/) (2026)
- [TRRG: Towards Truthful Radiology Report Generation With Cross-modal Disease Clue Enhanced Large Language Model](https://mlsanity.com/paper/2408.12141/) (2024)
- [A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography](https://mlsanity.com/paper/2502.05926/) (2025)
- [Anatomy-Aware Conditional Image-Text Retrieval](https://mlsanity.com/paper/2503.07456/) (2025)
- [Cross-Modal Clustering-Guided Negative Sampling for Self-Supervised Joint Learning from Medical Images and Reports](https://mlsanity.com/paper/2506.11674/) (2025)
- [SURE-Med: Systematic Uncertainty Reduction for Enhanced Reliability in Medical Report Generation](https://mlsanity.com/paper/2508.01693/) (2025)
- [Rad-VLSM: A Cross-Modal Framework with Semantics-Assisted Prompting for Medical Segmentation and Diagnosis](https://mlsanity.com/paper/2605.18130/) (2026)
- [Cross-modal linkage risk in clinical vision-language models](https://mlsanity.com/paper/2606.02276/) (2026)
- [EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models](https://mlsanity.com/paper/2606.06379/) (2026)
- [MAC-RRG: Iterative Multi-Agent Collaboration for X-ray Radiology Report Generation](https://mlsanity.com/paper/2609.26124/) (2026)
- [PLRS-IC: A Dual-Calibration Framework for Chest X-Ray Vision-Language Alignment](https://mlsanity.com/paper/2609.39266/) (2026)
- [CliQ-RRG: Clinical-Knowledge Guided Disease-aware Visual-Textual Alignment for QA-Style Radiology Report Generation](https://mlsanity.com/paper/2610.13059/) (2026)

## Abstract (arXiv)

To address these issues, we propose a novel Adaptive patch-word Matching (AdaMatch) model to correlate chest X-ray (CXR) image regions with words in medical reports and apply it to CXR-report generation to provide explainability for the generation process. AdaMatch exploits the fine-grained relation between adaptive patches and words to provide explanations of specific image regions with corresponding words. To capture the abnormal regions of varying sizes and positions, we introduce the Adaptive Patch extraction (AdaPatch) module to acquire the adaptive patches for these regions adaptively. In order to provide explicit explainability for CXR-report generation task, we propose an AdaMatch-based bidirectional large language model for Cyclic CXR-report generation (AdaMatch-Cyclic). It employs the AdaMatch to obtain the keywords for CXR images and `keypatches' for medical reports as hints to guide CXR-report generation. Extensive experiments on two publicly available CXR datasets prove the effectiveness of our method and its superior performance to existing methods.