# Lunguage: A Benchmark for Structured and Sequential Chest X-ray Interpretation

arXiv: https://arxiv.org/abs/2505.21190 · HTML: https://mlsanity.com/paper/2505.21190/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 25 of 25)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [MAIRA-2: Grounded Radiology Report Generation](https://mlsanity.com/paper/2406.04449/) (2024)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [GREEN: Generative Radiology Report Evaluation and Error Notation](https://mlsanity.com/paper/2405.03595/) (2024)
- [Improving Chest X-Ray Report Generation by Leveraging Warm Starting](https://mlsanity.com/paper/2201.09405/) (2022)
- [Chest ImaGenome Dataset for Clinical Reasoning](https://mlsanity.com/paper/2108.00316/) (2021)
- [Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards](https://mlsanity.com/paper/2210.12186/) (2022)
- [A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation](https://mlsanity.com/paper/2401.12208/) (2024)
- [RaTEScore: A Metric for Radiology Report Generation](https://mlsanity.com/paper/2406.16845/) (2024)
- [Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation](https://mlsanity.com/paper/2403.08002/) (2024)
- [MedVersa: A Generalist Foundation Model for Medical Image Interpretation](https://mlsanity.com/paper/2405.07988/) (2024)
- [Libra: Leveraging Temporal Images for Biomedical Radiology Analysis](https://mlsanity.com/paper/2411.19378/) (2024)
- [An Empirical Study of Qwen3 Quantization](https://mlsanity.com/paper/2505.02214/) (2025)
- [FineRadScore: A Radiology Report Line-by-Line Evaluation Technique Generating Corrections with Severity Scores](https://mlsanity.com/paper/2405.20613/) (2024)
- [RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extraction](https://mlsanity.com/paper/2308.05046/) (2023)
- [ReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reports](https://mlsanity.com/paper/2409.10829/) (2024)
- [Improving Radiology Report Conciseness and Structure via Local Large Language Models](https://mlsanity.com/paper/2411.05042/) (2024)

## Built on by (most cited first, 8 of 8)

- [Modeling Clinical Uncertainty in Radiology Reports: from Explicit Uncertainty Markers to Implicit Reasoning Pathways](https://mlsanity.com/paper/2511.04506/) (2025)
- [A Review of Longitudinal Radiology Report Generation: Dataset Composition, Methods, and Performance Evaluation](https://mlsanity.com/paper/2510.12444/) (2025)
- [MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation](https://mlsanity.com/paper/2508.16674/) (2025)
- [MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays](https://mlsanity.com/paper/2605.15574/) (2026)
- [CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rays](https://mlsanity.com/paper/2606.21020/) (2026)
- [Clinically-Grounded Hierarchical Classification for Consistent Chest X-ray Interpretation](https://mlsanity.com/paper/2608.03016/) (2026)
- [MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models](https://mlsanity.com/paper/2608.24118/) (2026)
- [Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation](https://mlsanity.com/paper/2609.19093/) (2026)

## Abstract (arXiv)

Radiology reports convey detailed clinical observations and capture diagnostic reasoning that evolves over time. However, existing evaluation methods are limited to single-report settings and rely on coarse metrics that fail to capture fine-grained clinical semantics and temporal dependencies. We introduce LUNGUAGE, a benchmark dataset for structured radiology report generation that supports both single-report evaluation and longitudinal patient-level assessment across multiple studies. It contains 1,473 annotated chest X-ray reports, each reviewed by experts, and 186 of them contain longitudinal annotations to capture disease progression and inter-study intervals, also reviewed by experts. Using this benchmark, we develop a two-stage structuring framework that transforms generated reports into fine-grained, schema-aligned structured reports, enabling longitudinal interpretation. We also propose LUNGUAGESCORE, an interpretable metric that compares structured outputs at the entity, relation, and attribute level while modeling temporal consistency across patient timelines. These contributions establish the first benchmark dataset, structuring framework, and evaluation metric for sequential radiology reporting, with empirical results demonstrating that LUNGUAGESCORE effectively supports structured report evaluation. The code is available at: https://github.com/SuperSupermoon/Lunguage