# Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation

arXiv: https://arxiv.org/abs/2501.03458 · HTML: https://mlsanity.com/paper/2501.03458/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 56 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [InternLM2 Technical Report](https://mlsanity.com/paper/2403.17297/) (2024)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [Orca 2: Teaching Small Language Models How to Reason](https://mlsanity.com/paper/2311.11045/) (2023)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats](https://mlsanity.com/paper/2405.19538/) (2024)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation](https://mlsanity.com/paper/2308.12604/) (2023)
- [Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignment](https://mlsanity.com/paper/2112.15011/) (2021)
- [Knowledge Matters: Radiology Report Generation with General and Specific Knowledge](https://mlsanity.com/paper/2112.15009/) (2021)
- [KiUT: Knowledge-injected U-Transformer for Radiology Report Generation](https://mlsanity.com/paper/2306.11345/) (2023)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning](https://mlsanity.com/paper/2306.06466/) (2023)
- [MedVersa: A Generalist Foundation Model for Medical Image Interpretation](https://mlsanity.com/paper/2405.07988/) (2024)
- [ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs](https://mlsanity.com/paper/2305.15964/) (2023)
- [Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports](https://mlsanity.com/paper/2306.08749/) (2023)
- [CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset](https://mlsanity.com/paper/2410.00379/) (2024)
- [HERGen: Elevating Radiology Report Generation with Longitudinal Data](https://mlsanity.com/paper/2407.15158/) (2024)
- [Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation](https://mlsanity.com/paper/2312.08078/) (2023)
- [Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation](https://mlsanity.com/paper/2207.01208/) (2022)

## Built on by (most cited first, 13 of 13)

- [Sign Language Translation using Frame and Event Stream: Benchmark Dataset and Algorithms](https://mlsanity.com/paper/2503.06484/) (2025)
- [R2GenKG: Hierarchical Multi-modal Knowledge Graph for LLM-based Radiology Report Generation](https://mlsanity.com/paper/2508.03426/) (2025)
- [RANGER: Sparsely-Gated Mixture-of-Experts with Adaptive Retrieval Re-ranking for Pathology Report Generation](https://mlsanity.com/paper/2603.04348/) (2026)
- [XiHeFusion: Harnessing Large Language Models for Science Communication in Nuclear Fusion](https://mlsanity.com/paper/2502.05615/) (2025)
- [Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation](https://mlsanity.com/paper/2605.01144/) (2026)
- [CogRad: A Cognitively-Inspired Multi-Agent Framework for Radiology Report Generation](https://mlsanity.com/paper/2607.03853/) (2026)
- [LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation](https://mlsanity.com/paper/2602.04617/) (2026)
- [Rethinking the Efficiency and Effectiveness of Reinforcement Learning for Radiology Report Generation](https://mlsanity.com/paper/2603.04022/) (2026)
- [PFM-VEPAR: Prompting Foundation Models for RGB-Event Camera based Pedestrian Attribute Recognition](https://mlsanity.com/paper/2603.19565/) (2026)
- [GTA-Net: Cooperative Game Theory for Vision-Language Alignment in Chest X-Ray Report Generation](https://mlsanity.com/paper/2606.21915/) (2026)
- [REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2607.10147/) (2026)
- [Homogeneous Semantic Alignment and Hierarchical Expert Routing for Radiology Report Generation](https://mlsanity.com/paper/2610.04499/) (2026)
- [Pre-training, Reasoning, Benchmarking: X-ray Report Generation on CheXpert Plus Dataset](https://mlsanity.com/paper/2610.08813/) (2026)

## Abstract (arXiv)

X-ray image based medical report generation achieves significant progress in recent years with the help of the large language model, however, these models have not fully exploited the effective information in visual image regions, resulting in reports that are linguistically sound but insufficient in describing key diseases. In this paper, we propose a novel associative memory-enhanced X-ray report generation model that effectively mimics the process of professional doctors writing medical reports. It considers both the mining of global and local visual information and associates historical report information to better complete the writing of the current report. Specifically, given an X-ray image, we first utilize a classification model along with its activation maps to accomplish the mining of visual regions highly associated with diseases and the learning of disease query tokens. Then, we employ a visual Hopfield network to establish memory associations for disease-related tokens, and a report Hopfield network to retrieve report memory information. This process facilitates the generation of high-quality reports based on a large language model and achieves state-of-the-art performance on multiple benchmark datasets, including the IU X-ray, MIMIC-CXR, and Chexpert Plus. The source code of this work is released on urlhttps://github.com/Event-AHU/MedicalImageAnalysis.