mlsanityData to 13 Oct 2026

Cross-modal Prototype Driven Network for Radiology Report Generation

arXiv 2207.04818 ↗First posted 11 Jul 2022cs.CV cs.CLcited by 59

Not on the map. Why?

Abstract (arXiv)

Radiology report generation (RRG) aims to describe automatically a radiology image with human-like language and could potentially support the work of radiologists, reducing the burden of manual reporting. Previous approaches often adopt an encoder-decoder architecture and focus on single-modal feature learning, while few studies explore cross-modal feature interaction. Here we propose a Cross-modal PROtotype driven NETwork (XPRONET) to promote cross-modal pattern learning and exploit it to improve the task of radiology report generation. This is achieved by three well-designed, fully differentiable and complementary modules: a shared cross-modal prototype matrix to record the cross-modal prototypes; a cross-modal prototype network to learn the cross-modal prototypes and embed the cross-modal information into the visual and textual features; and an improved multi-label contrastive loss to enable and enhance multi-label prototype learning. XPRONET obtains substantial improvements on the IU-Xray and MIMIC-CXR benchmarks, where its performance exceeds recent state-of-the-art approaches by a large margin on IU-Xray and comparable performance on MIMIC-CXR.

Key facts. Cited by 59 papers in the map (data through 13 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 11 Jul 2022.

Builds on

33 of 48 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 593
  8. 2016Self-critical Sequence Training for Image Captioningcited by 544
  9. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  10. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 332
  11. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  12. 2016Image Captioning with Semantic Attentioncited by 260
21 more
  1. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  2. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  3. 2020X-Linear Attention Networks for Image Captioningcited by 198
  4. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 184
  5. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  6. 2020When Radiology Report Generation Meets Knowledge Graphcited by 144
  7. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  8. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 140
  9. 2019Clinically Accurate Chest X-Ray Report Generationcited by 139
  10. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  11. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 102
  12. 2018Improving the Transformer Translation Model with Document-Level Contextcited by 74
  13. 2019Memory-Attended Recurrent Network for Video Captioningcited by 65
  14. 2020Normalized and Geometry-Aware Self-Attention Network for Image Captioningcited by 63
  15. 2020Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reportscited by 58
  16. 2020Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Networkcited by 48
  17. 2021G-Transformer for Document-level Machine Translationcited by 46
  18. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  19. 2021RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reportingcited by 34
  20. 2021Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networkscited by 24
  21. 2018simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captionscited by 20

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 21; 2025: 16; 2026: 11.

59 papers.

Sort:

By area:

  1. 2023ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Modelscited by 116
  2. 2024CT2Rep: Automated Radiology Report Generation for 3D Medical Imagingcited by 96
  3. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 62
  4. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  5. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  6. 2023Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reportscited by 35
  7. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 35
  8. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 33
  9. 2022CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generationcited by 22
  10. 2023Complex Organ Mask Guided Radiology Report Generationcited by 22
  11. 2023Cross-Modal Causal Intervention for Medical Report Generationcited by 15
  12. 2023ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Datacited by 14
38 more
  1. 2023A Systematic Review of Deep Learning-based Research on Radiology Report Generationcited by 14
  2. 2025Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generationcited by 13
  3. 2024Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generationcited by 11
  4. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  5. 2023X-TRA: Improving Chest X-ray Tasks with Cross-Modal Retrieval Augmentationcited by 10
  6. 2024Factual Serialization Enhancement: A Key Innovation for Chest X-ray Report Generationcited by 10
  7. 2024MedRAT: Unpaired Medical Report Generation via Auxiliary Taskscited by 9
  8. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  9. 2024A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Datacited by 6
  10. 2025Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuningcited by 5
  11. 2025R2GenKG: Hierarchical Multi-modal Knowledge Graph for LLM-based Radiology Report Generationcited by 4
  12. 2025A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generationcited by 4
  13. 2025MRG-R1: Reinforcement Learning for Clinically Aligned Medical Report Generationcited by 4
  14. 2024TiBiX: Leveraging Temporal Information for Bidirectional X-ray and Report Generationcited by 3
  15. 2024Learning To Guide Human Decision Makers With Vision-Language Modelscited by 3
  16. 2024Image-aware Evaluation of Generated Medical Reportscited by 3
  17. 2024ORID: Organ-Regional Information Driven Framework for Radiology Report Generationcited by 3
  18. 2025EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generationcited by 3
  19. 2023Automatic Radiology Report Generation by Learning with Increasingly Hard Negativescited by 2
  20. 2023Can Prompt Learning Benefit Radiology Report Generation?cited by 2
  21. 2024MedCycle: Unpaired Medical Report Generation via Cycle-Consistencycited by 2
  22. 2024Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Raycited by 2
  23. 2025MoColl: Agent-Based Specific and General Model Collaboration for Image Captioningcited by 2
  24. 2025Automated Structured Radiology Report Generation with Rich Clinical Contextcited by 2
  25. 2026AnchorDiff: Topology-Aware Masked Diffusion with Confidence-based Rewriting for Radiology Report Generationcited by 2
  26. 2024Dynamic Traceback Learning for Medical Report Generationcited by 1
  27. 2024Scene Graph Aided Radiology Report Generationcited by 1
  28. 2024Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion modelcited by 1
  29. 2024DAug: Diffusion-based Channel Augmentation for Radiology Image Retrieval and Classificationcited by 1
  30. 2025Med-R2: Perception and Reflection-driven Complex Reasoning for Medical Report Generationcited by 1
  31. 2025HSENet: Hybrid Spatial Encoding Network for 3D Medical Vision-Language Understandingcited by 1
  32. 2025Improving Medical Visual Representation Learning with Pathological-level Cross-Modal Alignment and Correlation Explorationcited by 1
  33. 2025OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generationcited by 1
  34. 2026GFPL: Generative Federated Prototype Learning for Resource-Constrained and Data-Imbalanced Vision Taskcited by 1
  35. 2026NeuroSymb-MRG: Differentiable Abductive Reasoning with Active Uncertainty Minimization for Radiology Report Generationcited by 1
  36. 2026RIHA: Report-Image Hierarchical Alignment for Radiology Report Generationcited by 1
  37. 2025UniCrossAdapter: Multimodal Adaptation of CLIP for Radiology Report Generationcited by 0
  38. 2025Advanced Chest X-Ray Analysis via Transformer-Based Image Descriptors and Cross-Model Attention Mechanismcited by 0

50 most cited of 59 All 59 (JSON)