# S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Framework

arXiv: https://arxiv.org/abs/2508.02082 · HTML: https://mlsanity.com/paper/2508.02082/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [R2GenGPT: Radiology Report Generation with Frozen LLMs](https://mlsanity.com/paper/2309.09812/) (2023)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)
- [RaTEScore: A Metric for Radiology Report Generation](https://mlsanity.com/paper/2406.16845/) (2024)
- [A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis](https://mlsanity.com/paper/2310.20381/) (2023)
- [Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting](https://mlsanity.com/paper/2307.05766/) (2023)
- [Replace and Report: NLP Assisted Radiology Report Generation](https://mlsanity.com/paper/2306.17180/) (2023)
- [KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models](https://mlsanity.com/paper/2409.05370/) (2024)
- [MRScore: Evaluating Radiology Report Generation with LLM-based Reward System](https://mlsanity.com/paper/2404.17778/) (2024)
- [FlexR: Few-shot Classification with Language Embeddings for Structured Reporting of Chest X-rays](https://mlsanity.com/paper/2203.15723/) (2022)

## Built on by (most cited first, 6 of 6)

- [CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation](https://mlsanity.com/paper/2603.06183/) (2026)
- [Multi-Modal Multi-Agent Reinforcement Learning for Radiology Report Generation](https://mlsanity.com/paper/2603.16876/) (2026)
- [Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance](https://mlsanity.com/paper/2606.18613/) (2026)
- [DRRG: A Discrete Diffusion Framework for Radiology Report Generation](https://mlsanity.com/paper/2608.24105/) (2026)
- [Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models](https://mlsanity.com/paper/2608.24121/) (2026)
- [EviDent-CBCT: Evidence-Bottlenecked Report Generation from Dental CBCT under Non-Exhaustive Report Supervision](https://mlsanity.com/paper/2610.02375/) (2026)

## Abstract (arXiv)

Radiology report generation (RRG) for diagnostic images, such as chest X-rays, plays a pivotal role in both clinical practice and AI. Traditional free-text reports suffer from redundancy and inconsistent language, complicating the extraction of critical clinical details. Structured radiology report generation (S-RRG) offers a promising solution by organizing information into standardized, concise formats. However, existing approaches often rely on classification or visual question answering (VQA) pipelines that require predefined label sets and produce only fragmented outputs. Template-based approaches, which generate reports by replacing keywords within fixed sentence patterns, further compromise expressiveness and often omit clinically important details. In this work, we present a novel approach to S-RRG that includes dataset construction, model training, and the introduction of a new evaluation framework. We first create a robust chest X-ray dataset (MIMIC-STRUC) that includes disease names, severity levels, probabilities, and anatomical locations, ensuring that the dataset is both clinically relevant and well-structured. We train an LLM-based model to generate standardized, high-quality reports. To assess the generated reports, we propose a specialized evaluation metric (S-Score) that not only measures disease prediction accuracy but also evaluates the precision of disease-specific details, thus offering a clinically meaningful metric for report quality that focuses on elements critical to clinical decision-making and demonstrates a stronger alignment with human assessments. Our approach highlights the effectiveness of structured reports and the importance of a tailored evaluation metric for S-RRG, providing a more clinically relevant measure of report quality.