mlsanityData to 14 Oct 2026

Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation

arXiv 2106.06963 ↗First posted 13 Jun 2021cs.CV cs.CLcited by 162

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Automatically generating radiology reports can improve current clinical practice in diagnostic radiology. On one hand, it can relieve radiologists from the heavy burden of report writing; On the other hand, it can remind radiologists of abnormalities and avoid the misdiagnosis and missed diagnosis. Yet, this task remains a challenging job for data-driven neural networks, due to the serious visual and textual data biases. To this end, we propose a Posterior-and-Prior Knowledge Exploring-and-Distilling approach (PPKED) to imitate the working patterns of radiologists, who will first examine the abnormal regions and assign the disease topic tags to the abnormal regions, and then rely on the years of prior medical knowledge and prior working experience accumulations to write reports. Thus, the PPKED includes three modules: Posterior Knowledge Explorer (PoKE), Prior Knowledge Explorer (PrKE) and Multi-domain Knowledge Distiller (MKD). In detail, PoKE explores the posterior knowledge, which provides explicit abnormal visual regions to alleviate visual data bias; PrKE explores the prior knowledge from the prior medical knowledge graph (medical knowledge) and prior radiology reports (working experience) to alleviate textual data bias. The explored knowledge is distilled by the MKD to generate the final reports. Evaluated on MIMIC-CXR and IU-Xray datasets, our method is able to outperform previous state-of-the-art models on these two datasets.

Key facts. Cited by 162 papers in the map (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 13 Jun 2021.

Builds on

32 of 48 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2016Layer Normalizationcited by 5,669
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  11. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
20 more
  1. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  2. 2016Self-critical Sequence Training for Image Captioningcited by 547
  3. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  4. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  5. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  6. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  7. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  8. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  9. 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
  10. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 102
  11. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  12. 2016Learning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotationcited by 59
  13. 2020Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reportscited by 58
  14. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  15. 2019Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representationscited by 36
  16. 2020Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generationcited by 35
  17. 2017Recurrent Topic-Transition GAN for Visual Paragraph Generationcited by 25
  18. 2018simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captionscited by 20
  19. 2020Exploring and Distilling Cross-Modal Information for Image Captioningcited by 19
  20. 2019Non-Autoregressive Coarse-to-Fine Video Captioningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 25; 2023: 42; 2024: 42; 2025: 26; 2026: 23.

162 papers.

Sort:

By area:

  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2023BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Taskscited by 201
  3. 2023MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiologycited by 149
  4. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 140
  5. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  6. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 124
  7. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 118
  8. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 114
  9. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 101
  10. 2022Improving Chest X-Ray Report Generation by Leveraging Warm Startingcited by 95
  11. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
  12. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 83
38 more
  1. 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 76
  2. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 73
  3. 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
  4. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  5. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 62
  6. 2022Cross-modal Prototype Driven Network for Radiology Report Generationcited by 59
  7. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  8. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 44
  9. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  10. 2023ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoningcited by 42
  11. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  12. 2023Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reportscited by 35
  13. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 35
  14. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 33
  15. 2022Explainable Deep Learning Methods in Medical Image Classification: A Surveycited by 32
  16. 2023Cascaded Latent Diffusion Models for High-Resolution Chest X-ray Synthesiscited by 32
  17. 2022SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Textcited by 30
  18. 2023Style-Aware Radiology Report Generation with RadGraph and Few-Shot Promptingcited by 29
  19. 2024MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generationcited by 29
  20. 2024CoMT: Chain-of-Medical-Thought Reduces Hallucination in Medical Report Generationcited by 25
  21. 2022Cross-modal Clinical Graph Transformer for Ophthalmic Report Generationcited by 24
  22. 2023Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generationcited by 23
  23. 2023Controllable Chest X-Ray Report Generation from Longitudinal Representationscited by 23
  24. 2024MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learningcited by 23
  25. 2022CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generationcited by 22
  26. 2023Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-trainingcited by 22
  27. 2023RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoningcited by 22
  28. 2023Complex Organ Mask Guided Radiology Report Generationcited by 22
  29. 2024PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imagingcited by 22
  30. 2024Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"cited by 20
  31. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  32. 2024Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directionscited by 18
  33. 2024Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretationcited by 18
  34. 2022Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generationcited by 17
  35. 2022A survey on attention mechanisms for medical applications: are we moving towards better algorithms?cited by 15
  36. 2022A Self-Guided Framework for Radiology Report Generationcited by 15
  37. 2023Cross-Modal Causal Intervention for Medical Report Generationcited by 15
  38. 2022RepsNet: Combining Vision with Language for Automated Medical Reportscited by 14

50 most cited of 162 All 162 (JSON)