# AutoRG-Brain: Grounded Report Generation for Brain MRI

arXiv: https://arxiv.org/abs/2407.16684 · HTML: https://mlsanity.com/paper/2407.16684/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 23 Jul 2024.

## Builds on (most cited first, 30 of 34)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images](https://mlsanity.com/paper/2201.01266/) (2022)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data](https://mlsanity.com/paper/2308.02463/) (2023)
- [Clinically Accurate Chest X-Ray Report Generation](https://mlsanity.com/paper/1904.02633/) (2019)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [Natural Synthetic Anomalies for Self-Supervised Anomaly Detection and Localization](https://mlsanity.com/paper/2109.15222/) (2021)
- [Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge](https://mlsanity.com/paper/1904.00682/) (2019)
- [ISLES 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset](https://mlsanity.com/paper/2206.06694/) (2022)
- [RaTEScore: A Metric for Radiology Report Generation](https://mlsanity.com/paper/2406.16845/) (2024)
- [Large-Vocabulary Segmentation for Medical Images with Text Prompts](https://mlsanity.com/paper/2312.17183/) (2023)
- [Label-Free Liver Tumor Segmentation](https://mlsanity.com/paper/2303.14869/) (2023)
- [The ASNR-MICCAI Brain Tumor Segmentation (BraTS) Challenge 2023: Intracranial Meningioma](https://mlsanity.com/paper/2305.07642/) (2023)
- [The Brain Tumor Segmentation (BraTS) Challenge 2023: Glioma Segmentation in Sub-Saharan Africa Patient Population (BraTS-Africa)](https://mlsanity.com/paper/2305.19369/) (2023)
- [The Brain Tumor Segmentation (BraTS) Challenge 2023: Focus on Pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs)](https://mlsanity.com/paper/2305.17033/) (2023)
- [Detecting Outliers with Poisson Image Interpolation](https://mlsanity.com/paper/2107.02622/) (2021)
- [DARE: Data Augmented Relation Extraction with GPT-2](https://mlsanity.com/paper/2004.13845/) (2020)
- [Progressive Transformer-Based Generation of Radiology Reports](https://mlsanity.com/paper/2102.09777/) (2021)
- [RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis](https://mlsanity.com/paper/2404.16754/) (2024)
- [The Brain Tumor Segmentation (BraTS-METS) Challenge 2023: Brain Metastasis Segmentation on Pre-treatment MRI](https://mlsanity.com/paper/2306.00838/) (2023)
- [Constrained unsupervised anomaly segmentation](https://mlsanity.com/paper/2203.01671/) (2022)

## Built on by (most cited first, 24 of 24)

- [Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2410.18387/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering](https://mlsanity.com/paper/2505.18915/) (2025)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Performance of GPT-5 in Brain Tumor MRI Reasoning](https://mlsanity.com/paper/2508.10865/) (2025)
- [A Clinically-Grounded Two-Stage Framework for Renal CT Report Generation](https://mlsanity.com/paper/2506.23584/) (2025)
- [Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric Approach](https://mlsanity.com/paper/2506.03238/) (2025)
- [Recurrent Visual Feature Extraction and Stereo Attentions for CT Report Generation](https://mlsanity.com/paper/2506.19665/) (2025)
- [Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRI](https://mlsanity.com/paper/2509.25889/) (2025)
- [Benchmarking the Thinking Mode of Multimodal Large Language Models in Clinical Tasks](https://mlsanity.com/paper/2511.03328/) (2025)
- [Vision-Language Models for Automated 3D PET/CT Report Generation](https://mlsanity.com/paper/2511.20145/) (2025)
- [BTReport: A Framework for Brain Tumor Radiology Report Generation with Clinically Relevant Features](https://mlsanity.com/paper/2602.16006/) (2026)
- [M^3Builder: A Multi-Agent System for Automated Machine Learning in Medical Imaging](https://mlsanity.com/paper/2502.20301/) (2025)
- [Automatic Fine-grained Segmentation-assisted Report Generation](https://mlsanity.com/paper/2507.16623/) (2025)
- [AMRG: Extend Vision Language Models for Automatic Mammography Report Generation](https://mlsanity.com/paper/2508.09225/) (2025)
- [Multimodal MRI Report Findings Supervised Brain Lesion Segmentation with Substructures](https://mlsanity.com/paper/2602.20994/) (2026)
- [3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection](https://mlsanity.com/paper/2602.23652/) (2026)
- [NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding](https://mlsanity.com/paper/2605.20525/) (2026)
- [TextSLIP: Text Self-Supervised CLIP for Medical Report Generation](https://mlsanity.com/paper/2607.21970/) (2026)
- [PerFact: Perception-Derived Fact Prompting for 3D Brain MRI Report Generation](https://mlsanity.com/paper/2608.17926/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)
- [BrainDiff: Longitudinal Report Generation for Multimodal Brain MRI](https://mlsanity.com/paper/2609.00593/) (2026)
- [The Diagnosis a Reporter Leaves Unspoken: Surfacing Frozen Tumor Features for Brain-Tumor MRI Reporting](https://mlsanity.com/paper/2609.02411/) (2026)
- [BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning](https://mlsanity.com/paper/2610.06571/) (2026)

## Abstract (arXiv)

Radiologists are tasked with interpreting a large number of images in a daily base, with the responsibility of generating corresponding reports. This demanding workload elevates the risk of human error, potentially leading to treatment delays, increased healthcare costs, revenue loss, and operational inefficiencies. To address these challenges, we initiate a series of work on grounded Automatic Report Generation (AutoRG), starting from the brain MRI interpretation system, which supports the delineation of brain structures, the localization of anomalies, and the generation of well-organized findings. We make contributions from the following aspects, first, on dataset construction, we release a comprehensive dataset encompassing segmentation masks of anomaly regions and manually authored reports, termed as RadGenome-Brain MRI. This data resource is intended to catalyze ongoing research and development in the field of AI-assisted report generation systems. Second, on system design, we propose AutoRG-Brain, the first brain MRI report generation system with pixel-level grounded visual clues. Third, for evaluation, we conduct quantitative assessments and human evaluations of brain structure segmentation, anomaly localization, and report generation tasks to provide evidence of its reliability and accuracy. This system has been integrated into real clinical scenarios, where radiologists were instructed to write reports based on our generated findings and anomaly segmentation masks. The results demonstrate that our system enhances the report-writing skills of junior doctors, aligning their performance more closely with senior doctors, thereby boosting overall productivity.