# Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation

arXiv: https://arxiv.org/abs/2006.03744 · HTML: https://mlsanity.com/paper/2006.03744/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2020.

## Built on by (most cited first, 30 of 35)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [KiUT: Knowledge-injected U-Transformer for Radiology Report Generation](https://mlsanity.com/paper/2306.11345/) (2023)
- [Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation](https://mlsanity.com/paper/2111.04318/) (2021)
- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [Explainable Deep Learning Methods in Medical Image Classification: A Survey](https://mlsanity.com/paper/2205.04766/) (2022)
- [Dia-LLaMA: Towards Large Language Model-driven CT Report Generation](https://mlsanity.com/paper/2403.16386/) (2024)
- [Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting](https://mlsanity.com/paper/2310.17811/) (2023)
- [Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation](https://mlsanity.com/paper/2206.01988/) (2022)
- [Medical Vision Language Pretraining: A survey](https://mlsanity.com/paper/2312.06224/) (2023)
- [A Systematic Review of Deep Learning-based Research on Radiology Report Generation](https://mlsanity.com/paper/2311.14199/) (2023)
- [Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining](https://mlsanity.com/paper/2304.14204/) (2023)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [Trust It or Not: Confidence-Guided Automatic Radiology Report Generation](https://mlsanity.com/paper/2106.10887/) (2021)
- [DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation](https://mlsanity.com/paper/2504.11786/) (2025)
- [Contrastive Learning with Counterfactual Explanations for Radiology Report Generation](https://mlsanity.com/paper/2407.14474/) (2024)
- [Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reports](https://mlsanity.com/paper/2401.16578/) (2024)
- [A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Data](https://mlsanity.com/paper/2405.12833/) (2024)
- [Topicwise Separable Sentence Retrieval for Medical Report Generation](https://mlsanity.com/paper/2405.04175/) (2024)
- [CrowdAgent: Multi-Agent Managed Multi-Source Annotation System](https://mlsanity.com/paper/2509.14030/) (2025)
- [Textual Inversion and Self-supervised Refinement for Radiology Report Generation](https://mlsanity.com/paper/2405.20607/) (2024)
- [Medical Report Generation Is A Multi-label Classification Problem](https://mlsanity.com/paper/2409.00250/) (2024)
- [A Labeled Ophthalmic Ultrasound Dataset with Medical Report Generation Based on Cross-modal Deep Learning](https://mlsanity.com/paper/2407.18667/) (2024)
- [MicarVLMoE: A Modern Gated Cross-Aligned Vision-Language Mixture of Experts Model for Medical Image Captioning and Report Generation](https://mlsanity.com/paper/2504.20343/) (2025)
- [Hallucination Mitigating for Medical Report Generation](https://mlsanity.com/paper/2601.15745/) (2026)

## Abstract (arXiv)

Beyond the common difficulties faced in the natural image captioning, medical report generation specifically requires the model to describe a medical image with a fine-grained and semantic-coherence paragraph that should satisfy both medical commonsense and logic. Previous works generally extract the global image features and attempt to generate a paragraph that is similar to referenced reports; however, this approach has two limitations. Firstly, the regions of primary interest to radiologists are usually located in a small area of the global image, meaning that the remainder parts of the image could be considered as irrelevant noise in the training procedure. Secondly, there are many similar sentences used in each medical report to describe the normal regions of the image, which causes serious data bias. This deviation is likely to teach models to generate these inessential sentences on a regular basis. To address these problems, we propose an Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) to mimic radiologists' working patterns. In more detail, ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning. The core structure of ASGK consists of a medical graph encoder and a natural language decoder, inspired by advanced Generative Pre-Training (GPT). Experiments on the CX-CHR dataset and our COVID-19 CT Report dataset demonstrate that our proposed ASGK is able to generate a robust and accurate report, and moreover outperforms state-of-the-art methods on both medical terminology classification and paragraph generation metrics.