# shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generation

arXiv: https://arxiv.org/abs/2306.03264 · HTML: https://mlsanity.com/paper/2306.03264/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards](https://mlsanity.com/paper/2210.12186/) (2022)
- [Learning to Summarize Radiology Findings](https://mlsanity.com/paper/1809.04698/) (2018)
- [Differentiable Multi-Agent Actor-Critic for Multi-Step Radiology Report Summarization](https://mlsanity.com/paper/2203.08257/) (2022)
- [FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarization](https://mlsanity.com/paper/2303.08335/) (2023)
- [RadLing: Towards Efficient Radiology Report Understanding](https://mlsanity.com/paper/2306.02492/) (2023)
- [Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Data](https://mlsanity.com/paper/2103.05131/) (2021)

## Built on by (most cited first, 7 of 7)

- [Large language models in medicine: the potentials and pitfalls](https://mlsanity.com/paper/2309.00087/) (2023)
- [Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domains](https://mlsanity.com/paper/2311.14966/) (2023)
- [Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering](https://mlsanity.com/paper/2404.16192/) (2024)
- [Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polish](https://mlsanity.com/paper/2402.09759/) (2024)
- [Natural Language Generation in Healthcare: A Review of Methods and Applications](https://mlsanity.com/paper/2505.04073/) (2025)
- [Generation of Radiology Findings in Chest X-Ray by Leveraging Collaborative Knowledge](https://mlsanity.com/paper/2306.10448/) (2023)
- [Less Is More? Selective Visual Attention to High-Importance Regions for Multimodal Radiology Summarization](https://mlsanity.com/paper/2603.29901/) (2026)

## Abstract (arXiv)

Instruction-tuned generative Large language models (LLMs) like ChatGPT and Bloomz possess excellent generalization abilities, but they face limitations in understanding radiology reports, particularly in the task of generating the IMPRESSIONS section from the FINDINGS section. They tend to generate either verbose or incomplete IMPRESSIONS, mainly due to insufficient exposure to medical text data during training. We present a system which leverages large-scale medical text data for domain-adaptive pre-training of instruction-tuned LLMs to enhance its medical knowledge and performance on specific medical tasks. We show that this system performs better in a zero-shot setting than a number of pretrain-and-finetune adaptation methods on the IMPRESSIONS generation task, and ranks 1st among participating systems in Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.