# RadLing: Towards Efficient Radiology Report Understanding

arXiv: https://arxiv.org/abs/2306.02492 · HTML: https://mlsanity.com/paper/2306.02492/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [Improving Biomedical Pretrained Language Models with Knowledge](https://mlsanity.com/paper/2104.10344/) (2021)
- [Question Answering is a Format; When is it Useful?](https://mlsanity.com/paper/1909.11291/) (2019)
- [AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain](https://mlsanity.com/paper/2110.13434/) (2021)
- [Differentiable Multi-Agent Actor-Critic for Multi-Step Radiology Report Summarization](https://mlsanity.com/paper/2203.08257/) (2022)
- [ICDBigBird: A Contextual Embedding Model for ICD Code Classification](https://mlsanity.com/paper/2204.10408/) (2022)

## Built on by (most cited first, 5 of 5)

- [shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generation](https://mlsanity.com/paper/2306.03264/) (2023)
- [Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering](https://mlsanity.com/paper/2404.16192/) (2024)
- [Generation of Radiology Findings in Chest X-Ray by Leveraging Collaborative Knowledge](https://mlsanity.com/paper/2306.10448/) (2023)
- [A Dual-View Approach to Classifying Radiology Reports by Co-Training](https://mlsanity.com/paper/2406.05995/) (2024)
- [Dissociating Answerability from Extraction in LLM Question Answering over Radiology Reports](https://mlsanity.com/paper/2610.17576/) (2026)

## Abstract (arXiv)

Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus. There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks. We present RadLing, a continuously pretrained language model using Electra-small (Clark et al., 2020) architecture, trained using over 500K radiology reports, that can compete with state-of-the-art results for fine tuning tasks in radiology domain. Our main contribution in this paper is knowledge-aware masking which is a taxonomic knowledge-assisted pretraining task that dynamically masks tokens to inject knowledge during pretraining. In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.