# CORAL: Expert-Curated medical Oncology Reports to Advance Language Model Inference

arXiv: https://arxiv.org/abs/2308.03853 · HTML: https://mlsanity.com/paper/2308.03853/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 18 references that are in the map.
- First posted on arXiv 7 Aug 2023.

## Builds on (most cited first, 5 of 5)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Learning from Task Descriptions](https://mlsanity.com/paper/2011.08115/) (2020)
- [A frame semantic overview of NLP-based information extraction for cancer-related EHR notes](https://mlsanity.com/paper/1904.01655/) (2019)

## Built on by (most cited first, 8 of 8)

- [Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education](https://mlsanity.com/paper/2409.18924/) (2024)
- [FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs](https://mlsanity.com/paper/2412.12422/) (2024)
- [A comparative study of zero-shot inference with large language models and supervised modeling in breast cancer pathology classification](https://mlsanity.com/paper/2401.13887/) (2024)
- [The Minimum Information about CLinical Artificial Intelligence Checklist for Generative Modeling Research (MI-CLAIM-GEN)](https://mlsanity.com/paper/2403.02558/) (2024)
- [Clinical Knowledge Graph Construction and Evaluation with Multi-LLMs via Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.01844/) (2026)
- [Embedding-Driven Diversity Sampling to Improve Few-Shot Synthetic Data Generation](https://mlsanity.com/paper/2501.11199/) (2025)
- [Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes](https://mlsanity.com/paper/2511.20680/) (2025)
- [MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction](https://mlsanity.com/paper/2605.20197/) (2026)

## Abstract (arXiv)

Both medical care and observational studies in oncology require a thorough understanding of a patient's disease progression and treatment history, often elaborately documented in clinical notes. Despite their vital role, no current oncology information representation and annotation schema fully encapsulates the diversity of information recorded within these notes. Although large language models (LLMs) have recently exhibited impressive performance on various medical natural language processing tasks, due to the current lack of comprehensively annotated oncology datasets, an extensive evaluation of LLMs in extracting and reasoning with the complex rhetoric in oncology notes remains understudied. We developed a detailed schema for annotating textual oncology information, encompassing patient characteristics, tumor characteristics, tests, treatments, and temporality. Using a corpus of 40 de-identified breast and pancreatic cancer progress notes at University of California, San Francisco, we applied this schema to assess the zero-shot abilities of three recent LLMs (GPT-4, GPT-3.5-turbo, and FLAN-UL2) to extract detailed oncological history from two narrative sections of clinical progress notes. Our team annotated 9028 entities, 9986 modifiers, and 5312 relationships. The GPT-4 model exhibited overall best performance, with an average BLEU score of 0.73, an average ROUGE score of 0.72, an exact-match F1-score of 0.51, and an average accuracy of 68% on complex tasks (expert manual evaluation on subset). Notably, it was proficient in tumor characteristic and medication extraction, and demonstrated superior performance in relational inference like adverse event detection. However, further improvements are needed before using it to reliably extract important facts from cancer progress notes needed for clinical research, complex population management, and documenting quality patient care.