mlsanityData to 14 Oct 2026

CUED at ProbSum 2023: Hierarchical Ensemble of Summarization Models

arXiv 2306.05317 ↗First posted 8 Jun 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider the challenge of summarizing patients' medical progress notes in a limited data setting. For the Problem List Summarization (shared task 1A) at the BioNLP Workshop 2023, we demonstrate that Clinical-T5 fine-tuned to 765 medical clinic notes outperforms other extractive, abstractive and zero-shot baselines, yielding reasonable baseline systems for medical note summarization. Further, we introduce Hierarchical Ensemble of Summarization Models (HESM), consisting of token-level ensembles of diverse fine-tuned Clinical-T5 models, followed by Minimum Bayes Risk (MBR) decoding. Our HESM approach lead to a considerable summarization performance boost, and when evaluated on held-out challenge data achieved a ROUGE-L of 32.77, which was the best-performing system at the top of the shared task leaderboard.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

24 of 37 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  10. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  12. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
12 more
  1. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  2. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  3. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  4. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 246
  5. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  6. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  7. 2023Do We Still Need Clinical Language Models?cited by 98
  8. 2017Ensemble Distillation for Neural Machine Translationcited by 39
  9. 2022Summarizing Patients Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Modelscited by 30
  10. 2021Long-Span Summarization via Local Attention and Content Selectioncited by 24
  11. 2020Ensemble Distillation Approaches for Grammatical Error Correctioncited by 8
  12. 2020CUED_speech at TREC 2020 Podcast Summarisation Trackcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 3.

8 papers.

  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  2. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 246
  3. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 51
  4. 2023Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Studycited by 30
  5. 2024Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLMcited by 27
  6. 2023Minimum Bayes' Risk Decoding for System Combination of Grammatical Error Correction Systemscited by 4
  7. 2024e-Health CSIRO at "Discharge Me!" 2024: Generating Discharge Summary Sections with Fine-tuned Language Modelscited by 4
  8. 2024Efficient Sample-Specific Encoder Perturbationscited by 0