# Evaluating AI Generated Summaries for Cancer Patients

arXiv: https://arxiv.org/abs/2608.26154 · HTML: https://mlsanity.com/paper/2608.26154/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 28 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 10 of 10)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Medical Hallucinations in Foundation Models and Their Impact on Healthcare](https://mlsanity.com/paper/2503.05777/) (2025)
- [Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI](https://mlsanity.com/paper/2311.01463/) (2023)
- [Attention-based Clinical Note Summarization](https://mlsanity.com/paper/2104.08942/) (2021)
- [Bias in Large Language Models Across Clinical Applications: A Systematic Review](https://mlsanity.com/paper/2504.02917/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly being integrated into digital health platforms to generate summaries of complex medical data. Although these models can improve patient engagement and communication, these systems also raise concerns about accuracy, faithfulness, and safety in clinical contexts. In this study, we evaluate AI-generated summaries within a cancer patient care application using a dual assessment framework. Human domain experts, including oncology clinicians and patient-facing care staff, provided ground-truth evaluations of summary quality along dimensions of accuracy, clinical relevance, and readability. In parallel, we employed LLMs serving as evaluators (LLM-as-a-judge). Some limitations were identified in the generated summaries e.g., occasional omissions and minor inaccuracies. These were systematically analyzed and used to iteratively improve prompt design, grounding, and safety guardrails.