# PULSAR: Pre-training with Extracted Healthcare Terms for Summarising Patients' Problems and Data Augmentation with Black-box Large Language Models

arXiv: https://arxiv.org/abs/2306.02754 · HTML: https://mlsanity.com/paper/2306.02754/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Learning to Summarize Radiology Findings](https://mlsanity.com/paper/1809.04698/) (2018)
- [Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations](https://mlsanity.com/paper/2109.12174/) (2021)
- [What's in a Summary? Laying the Groundwork for Advances in Hospital-Course Summarization](https://mlsanity.com/paper/2105.00816/) (2021)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [Ontology-Aware Clinical Abstractive Summarization](https://mlsanity.com/paper/1905.05818/) (2019)
- [Summarizing Patients Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models](https://mlsanity.com/paper/2208.08408/) (2022)
- [Attend to Medical Ontologies: Content Selection for Clinical Abstractive Summarization](https://mlsanity.com/paper/2005.00163/) (2020)
- [Hierarchical Annotation for Building A Suite of Clinical Natural Language Processing Tasks: Progress Note Understanding](https://mlsanity.com/paper/2204.03035/) (2022)
- [Do You Hear The People Sing? Key Point Analysis via Iterative Clustering and Abstractive Summarisation](https://mlsanity.com/paper/2305.16000/) (2023)

## Built on by (most cited first, 6 of 6)

- [Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study](https://mlsanity.com/paper/2308.14321/) (2023)
- [MIRA: Medical Time Series Foundation Model for Real-World Health Data](https://mlsanity.com/paper/2506.07584/) (2025)
- [TarDiff: Target-Oriented Diffusion Guidance for Synthetic Electronic Health Record Time Series Generation](https://mlsanity.com/paper/2504.17613/) (2025)
- [SynBench: A Benchmark for Differentially Private Text Generation](https://mlsanity.com/paper/2509.14594/) (2025)
- [Utilizing Large Language Models to Generate Synthetic Data to Increase the Performance of BERT-Based Neural Networks](https://mlsanity.com/paper/2405.06695/) (2024)
- [Generating Realistic Multi-Beat ECG Signals](https://mlsanity.com/paper/2505.18189/) (2025)

## Abstract (arXiv)

Medical progress notes play a crucial role in documenting a patient's hospital journey, including his or her condition, treatment plan, and any updates for healthcare providers. Automatic summarisation of a patient's problems in the form of a problem list can aid stakeholders in understanding a patient's condition, reducing workload and cognitive bias. BioNLP 2023 Shared Task 1A focuses on generating a list of diagnoses and problems from the provider's progress notes during hospitalisation. In this paper, we introduce our proposed approach to this task, which integrates two complementary components. One component employs large language models (LLMs) for data augmentation; the other is an abstractive summarisation LLM with a novel pre-training objective for generating the patients' problems summarised as a list. Our approach was ranked second among all submissions to the shared task. The performance of our model on the development and test datasets shows that our approach is more robust on unknown data, with an improvement of up to 3.1 points over the same size of the larger model.