mlsanityData to 7 Oct 2026

R2GenGPT: Radiology Report Generation with Frozen LLMs

arXiv 2309.09812 ↗First posted 18 Sep 2023cs.CVcited by 111

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have consistently showcased remarkable generalization capabilities when applied to various language tasks. Nonetheless, harnessing the full potential of LLMs for Radiology Report Generation (R2Gen) still presents a challenge, stemming from the inherent disparity in modality between LLMs and the R2Gen task. To bridge this gap effectively, we propose R2GenGPT, which is a novel solution that aligns visual features with the word embedding space of LLMs using an efficient visual alignment module. This innovative approach empowers the previously static LLM to seamlessly integrate and process image information, marking a step forward in optimizing R2Gen performance. R2GenGPT offers the following benefits. First, it attains state-of-the-art (SOTA) performance by training only the lightweight visual alignment module while freezing all the parameters of LLM. Second, it exhibits high training efficiency, as it requires the training of an exceptionally minimal number of parameters while achieving rapid convergence. By employing delta tuning, our model only trains 5M parameters (which constitute just 0.07% of the total parameter count) to achieve performance close to the SOTA levels. Our code is available at https://github.com/wang-zhanyu/R2GenGPT.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 40 of its 50 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

40 of 50 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
28 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  3. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  6. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  7. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  8. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  9. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  10. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  11. 2016Image Captioning with Semantic Attentioncited by 260
  12. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  13. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  14. 2020X-Linear Attention Networks for Image Captioningcited by 198
  15. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
  16. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 160
  17. 2020When Radiology Report Generation Meets Knowledge Graphcited by 144
  18. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
  19. 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 114
  20. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 101
  21. 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
  22. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 92
  23. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  24. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  25. 2023KiUT: Knowledge-injected U-Transformer for Radiology Report Generationcited by 60
  26. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  27. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 42
  28. 2021X-modaler: A Versatile and High-performance Codebase for Cross-modal Analyticscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 22; 2025: 49; 2026: 37.

111 papers.

Sort:

By area:

  1. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  2. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  3. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 44
  4. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 39
  5. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  6. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 33
  7. 2024VividMed: Vision Language Model with Versatile Visual Grounding for Medicinecited by 28
  8. 2024Large Language Model with Region-guided Referring and Grounding for CT Report Generationcited by 27
  9. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 23
  10. 2025CT-Agent: A Multimodal-LLM Agent for 3D CT Radiology Question Answeringcited by 20
  11. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  12. 2025Multimodal Large Language Models for Medicine: A Comprehensive Surveycited by 16
38 more
  1. 2023Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generationcited by 12
  2. 2025RadAlign: Advancing Radiology Report Generation with Vision-Language Concept Alignmentcited by 12
  3. 2026EHRWorld: A Patient-Centric Medical World Model for Long-Horizon Clinical Trajectoriescited by 12
  4. 2024R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generationcited by 11
  5. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  6. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  7. 2025Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generationcited by 11
  8. 2025RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injectioncited by 11
  9. 2024HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generationcited by 9
  10. 2025PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generationcited by 9
  11. 2025A Multimodal Multi-Agent Framework for Radiology Report Generationcited by 8
  12. 2025DDaTR: Dynamic Difference-aware Temporal Residual Network for Longitudinal Radiology Report Generationcited by 7
  13. 2025CXRAgent: Director-Orchestrated Multi-Stage Reasoning for Chest X-Ray Interpretationcited by 7
  14. 2024A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Datacited by 6
  15. 2024Clinical Insights: A Comprehensive Review of Language Models in Medicinecited by 6
  16. 2024KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Modelscited by 6
  17. 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
  18. 2025A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generationcited by 6
  19. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  20. 2025S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Frameworkcited by 5
  21. 2026XrayClaw: Cooperative-Competitive Multi-Agent Alignment for Trustworthy Chest X-ray Diagnosiscited by 5
  22. 2025Walking the Tightrope: Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuningcited by 4
  23. 2025Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuningcited by 4
  24. 2025LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learningcited by 4
  25. 2024Learning To Guide Human Decision Makers With Vision-Language Modelscited by 3
  26. 2025Reason Like a Radiologist: Chain-of-Thought and Reinforcement Learning for Verifiable Report Generationcited by 3
  27. 2025R2GenKG: Hierarchical Multi-modal Knowledge Graph for LLM-based Radiology Report Generationcited by 3
  28. 2025Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generationcited by 3
  29. 2025A Review of Longitudinal Radiology Report Generation: Dataset Composition, Methods, and Performance Evaluationcited by 3
  30. 2025EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generationcited by 3
  31. 2026CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generationcited by 3
  32. 2026MARCH: Multi-Agent Radiology Clinical Hierarchy for CT Report Generationcited by 3
  33. 2024Medical Report Generation Is A Multi-label Classification Problemcited by 2
  34. 2025CBM-RAG: Demonstrating Enhanced Interpretability in Radiology Report Generation with Multi-Agent RAG and Concept Bottleneck Modelscited by 2
  35. 2025Large-scale Multi-sequence Pretraining for Generalizable MRI Analysis in Versatile Clinical Applicationscited by 2
  36. 2025Phrase-grounded Fact-checking for Automatically Generated Chest X-ray Reportscited by 2
  37. 2025Automated Structured Radiology Report Generation with Rich Clinical Contextcited by 2
  38. 2025A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generationcited by 2

50 most cited of 111 All 111 (JSON)