mlsanityData to 14 Oct 2026

Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation

arXiv 1805.08298 ↗First posted 21 May 2018cs.CVcited by 141

Not on the map. Why?

Abstract (arXiv)

Generating long and coherent reports to describe medical images poses challenges to bridging visual patterns with informative human linguistic descriptions. We propose a novel Hybrid Retrieval-Generation Reinforced Agent (HRGR-Agent) which reconciles traditional retrieval-based approaches populated with human prior knowledge, with modern learning-based approaches to achieve structured, robust, and diverse report generation. HRGR-Agent employs a hierarchical decision-making procedure. For each sentence, a high-level retrieval policy module chooses to either retrieve a template sentence from an off-the-shelf template database, or invoke a low-level generation module to generate a new sentence. HRGR-Agent is updated via reinforcement learning, guided by sentence-level and word-level rewards. Experiments show that our approach achieves the state-of-the-art results on two medical report datasets, generating well-balanced structured sentences with robust coverage of heterogeneous medical report contents. In addition, our model achieves the highest detection accuracy of medical terminologies, and improved human evaluation performance.

Key facts. Cited by 141 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 27; 2022: 17; 2023: 34; 2024: 30; 2025: 18; 2026: 15.

141 papers.

Sort:

By area:

  1. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  2. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 185
  3. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  4. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 140
  5. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 118
  6. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 101
  7. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95
  8. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 90
  9. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 83
  10. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 76
  11. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 73
  12. 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
38 more
  1. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  2. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  3. 2023Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Reviewcited by 62
  4. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 62
  5. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 59
  6. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  7. 2023PRIOR: Prototype Representation Joint Learning from Medical Images and Reportscited by 55
  8. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  9. 2021Progressive Transformer-Based Generation of Radiology Reportscited by 43
  10. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  11. 2023ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoningcited by 42
  12. 2021Weakly Supervised Contrastive Learning for Chest X-Ray Report Generationcited by 41
  13. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  14. 2021What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarizationcited by 35
  15. 2023Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reportscited by 35
  16. 2022Explainable Deep Learning Methods in Medical Image Classification: A Surveycited by 32
  17. 2024MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generationcited by 29
  18. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  19. 2021Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learningcited by 24
  20. 2021Automated Generation of Accurate & Fluent Medical X-ray Reportscited by 24
  21. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23
  22. 2024Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generationcited by 23
  23. 2022CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generationcited by 22
  24. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-trainingcited by 22
  25. 2023RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoningcited by 22
  26. 2021AnaXNet: Anatomy Aware Multi-label Finding Classification in Chest X-raycited by 19
  27. 2021Writing by Memorizing: Hierarchical Retrieval-based Medical Report Generationcited by 19
  28. 2022Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generationcited by 17
  29. 2021Contextualized Keyword Representations for Multi-modal Retinal Image Captioningcited by 16
  30. 2021Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"cited by 16
  31. 2021Variational Topic Inference for Chest X-Ray Report Generationcited by 16
  32. 2024Radiology Report Generation via Multi-objective Preference Optimizationcited by 16
  33. 2022A Self-Guided Framework for Radiology Report Generationcited by 15
  34. 2022RepsNet: Combining Vision with Language for Automated Medical Reportscited by 14
  35. 2023Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generationcited by 14
  36. 2023A Systematic Review of Deep Learning-based Research on Radiology Report Generationcited by 14
  37. 2021Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignmentcited by 13
  38. 2022DeltaNet:Conditional Medical Report Generation for COVID-19 Diagnosiscited by 13

50 most cited of 141 All 141 (JSON)