# Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization

arXiv: https://arxiv.org/abs/2008.11293 · HTML: https://mlsanity.com/paper/2008.11293/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Aug 2020.

## Built on by (most cited first, 30 of 52)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [SciFact-Open: Towards open-domain scientific claim verification](https://mlsanity.com/paper/2210.13777/) (2022)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [MS2: Multi-Document Summarization of Medical Studies](https://mlsanity.com/paper/2104.06486/) (2021)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [Towards Interpretable Mental Health Analysis with Large Language Models](https://mlsanity.com/paper/2304.03347/) (2023)
- [Detection and Measurement of Syntactic Templates in Generated Text](https://mlsanity.com/paper/2407.00211/) (2024)
- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [Towards Evaluating and Building Versatile Large Language Models for Medicine](https://mlsanity.com/paper/2408.12547/) (2024)
- [Evaluating Factuality in Text Simplification](https://mlsanity.com/paper/2204.07562/) (2022)
- [A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?](https://mlsanity.com/paper/2409.15277/) (2024)
- [CHIME: LLM-Assisted Hierarchical Organization of Scientific Studies for Literature Review Support](https://mlsanity.com/paper/2407.16148/) (2024)
- [Retrieval augmentation of large language models for lay language generation](https://mlsanity.com/paper/2211.03818/) (2022)
- [Closing the gap between open-source and commercial large language models for medical evidence summarization](https://mlsanity.com/paper/2408.00588/) (2024)
- [Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval](https://mlsanity.com/paper/2212.10526/) (2022)
- [Readability Controllable Biomedical Document Summarization](https://mlsanity.com/paper/2210.04705/) (2022)
- [Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews](https://mlsanity.com/paper/2305.11828/) (2023)
- [Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations](https://mlsanity.com/paper/2305.13693/) (2023)
- [LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation](https://mlsanity.com/paper/2309.12455/) (2023)
- [Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization](https://mlsanity.com/paper/2311.09335/) (2023)
- [Accelerating Clinical Evidence Synthesis with Large Language Models](https://mlsanity.com/paper/2406.17755/) (2024)
- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [APPLS: Evaluating Evaluation Metrics for Plain Language Summarization](https://mlsanity.com/paper/2305.14341/) (2023)
- [Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domains](https://mlsanity.com/paper/2311.14966/) (2023)
- [A Survey on Medical Document Summarization](https://mlsanity.com/paper/2212.01669/) (2022)
- [Self-Repetition in Abstractive Neural Summarizers](https://mlsanity.com/paper/2210.08145/) (2022)
- [Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: A Benchmark Study](https://mlsanity.com/paper/2402.19052/) (2024)
- [Do Multi-Document Summarization Models Synthesize?](https://mlsanity.com/paper/2301.13844/) (2023)
- [A foundation model for human-AI collaboration in medical literature mining](https://mlsanity.com/paper/2501.16255/) (2025)
- [Panacea: A foundation model for clinical trial search, summarization, design, and recruitment](https://mlsanity.com/paper/2407.11007/) (2024)

## Abstract (arXiv)

We consider the problem of automatically generating a narrative biomedical evidence summary from multiple trial reports. We evaluate modern neural models for abstractive summarization of relevant article abstracts from systematic reviews previously conducted by members of the Cochrane collaboration, using the authors conclusions section of the review abstract as our target. We enlist medical professionals to evaluate generated summaries, and we find that modern summarization systems yield consistently fluent and relevant synopses, but that they are not always factual. We propose new approaches that capitalize on domain-specific models to inform summarization, e.g., by explicitly demarcating snippets of inputs that convey key findings, and emphasizing the reports of large and high-quality trials. We find that these strategies modestly improve the factual accuracy of generated summaries. Finally, we propose a new method for automatically evaluating the factuality of generated narrative evidence syntheses using models that infer the directionality of reported findings.