mlsanityData to 7 Oct 2026

RadLing: Towards Efficient Radiology Report Understanding

arXiv 2306.02492 ↗First posted 4 Jun 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Most natural language tasks in the radiology domain use language models pre-trained on biomedical corpus. There are few pretrained language models trained specifically for radiology, and fewer still that have been trained in a low data setting and gone on to produce comparable results in fine-tuning tasks. We present RadLing, a continuously pretrained language model using Electra-small (Clark et al., 2020) architecture, trained using over 500K radiology reports, that can compete with state-of-the-art results for fine tuning tasks in radiology domain. Our main contribution in this paper is knowledge-aware masking which is a taxonomic knowledge-assisted pretraining task that dynamically masks tokens to inject knowledge during pretraining. In addition, we also introduce an knowledge base-aided vocabulary extension to adapt the general tokenization vocabulary to radiology domain.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 21 of its 43 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

21 of 43 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  7. 2019Publicly Available Clinical BERT Embeddingscited by 995
  8. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  9. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  10. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  11. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  12. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
9 more
  1. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  2. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  3. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  4. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  5. 2021Improving Biomedical Pretrained Language Models with Knowledgecited by 36
  6. 2019Question Answering is a Format; When is it Useful?cited by 25
  7. 2021AVocaDo: Strategy for Adapting Vocabulary to Downstream Domaincited by 20
  8. 2022Differentiable Multi-Agent Actor-Critic for Multi-Step Radiology Report Summarizationcited by 13
  9. 2022ICDBigBird: A Contextual Embedding Model for ICD Code Classificationcited by 10

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2.

4 papers.

  1. 2023shs-nlp at RadSum23: Domain-Adaptive Pre-training of Instruction-tuned LLMs for Radiology Report Impression Generationcited by 7
  2. 2024Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answeringcited by 5
  3. 2023Generation of Radiology Findings in Chest X-Ray by Leveraging Collaborative Knowledgecited by 2
  4. 2024A Dual-View Approach to Classifying Radiology Reports by Co-Trainingcited by 0