# Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation

arXiv: https://arxiv.org/abs/2204.00447 · HTML: https://mlsanity.com/paper/2204.00447/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 1 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports](https://mlsanity.com/paper/1911.02541/) (2019)
- [Medically Aware GPT-3 as a Data Generator for Medical Dialogue Summarization](https://mlsanity.com/paper/2110.07356/) (2021)
- [Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniques](https://mlsanity.com/paper/2005.01795/) (2020)
- [Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation](https://mlsanity.com/paper/1706.09799/) (2017)
- [Learning to Summarize Radiology Findings](https://mlsanity.com/paper/1809.04698/) (2018)
- [Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures](https://mlsanity.com/paper/2009.08666/) (2020)
- [Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations](https://mlsanity.com/paper/2109.12174/) (2021)
- [Topic-aware Pointer-Generator Networks for Summarizing Spoken Conversations](https://mlsanity.com/paper/1910.01335/) (2019)
- [Ontology-Aware Clinical Abstractive Summarization](https://mlsanity.com/paper/1905.05818/) (2019)
- [Extractive Summarization of EHR Discharge Notes](https://mlsanity.com/paper/1810.12085/) (2018)
- [A preliminary study on evaluating Consultation Notes with Post-Editing](https://mlsanity.com/paper/2104.04402/) (2021)

## Built on by (most cited first, 30 of 33)

- [A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review](https://mlsanity.com/paper/2405.02559/) (2024)
- [A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization](https://mlsanity.com/paper/2303.03948/) (2023)
- [De-identification is not enough: a comparison between de-identified and synthetic clinical notes](https://mlsanity.com/paper/2402.00179/) (2024)
- [User-Driven Research of Medical Note Generation Software](https://mlsanity.com/paper/2205.02549/) (2022)
- [We Should Evaluate Real-World Impact](https://mlsanity.com/paper/2507.05973/) (2025)
- [Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generators](https://mlsanity.com/paper/2212.06008/) (2022)
- [An Investigation of Evaluation Metrics for Automated Medical Note Generation](https://mlsanity.com/paper/2305.17364/) (2023)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation](https://mlsanity.com/paper/2405.00715/) (2024)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation](https://mlsanity.com/paper/2211.09455/) (2022)
- [WangLab at MEDIQA-Chat 2023: Clinical Note Generation from Doctor-Patient Conversations using Large Language Models](https://mlsanity.com/paper/2305.02220/) (2023)
- [Improving Summarization with Human Edits](https://mlsanity.com/paper/2310.05857/) (2023)
- [Strong hallucinations from negation and how to fix them](https://mlsanity.com/paper/2402.10543/) (2024)
- [A Data-Centric Approach To Generate Faithful and High Quality Patient Summaries with Large Language Models](https://mlsanity.com/paper/2402.15422/) (2024)
- [The Pitfalls of Defining Hallucination](https://mlsanity.com/paper/2401.07897/) (2024)
- [Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questions](https://mlsanity.com/paper/2404.13307/) (2024)
- [Toward Relieving Clinician Burden by Automatically Generating Progress Notes using Interim Hospital Data](https://mlsanity.com/paper/2410.12845/) (2024)
- [A Survey for Biomedical Text Summarization: From Pre-trained to Large Language Models](https://mlsanity.com/paper/2304.08763/) (2023)
- [Enhancing Summarization Performance through Transformer-Based Prompt Engineering in Automated Medical Reporting](https://mlsanity.com/paper/2311.13274/) (2023)
- [Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review](https://mlsanity.com/paper/2409.18170/) (2024)
- [Automatic Logical Forms improve fidelity in Table-to-Text generation](https://mlsanity.com/paper/2310.17279/) (2023)
- [Are LLMs reliable? An exploration of the reliability of large language models in clinical note generation](https://mlsanity.com/paper/2505.17095/) (2025)
- [MedSyn: Enhancing Diagnostics with Human-AI Collaboration](https://mlsanity.com/paper/2506.14774/) (2025)
- [When Reasoning Hurts: Source-Aware Evaluation of Frontier LLMs for Clinical SOAP Note Generation](https://mlsanity.com/paper/2605.24902/) (2026)
- [Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations](https://mlsanity.com/paper/2311.13273/) (2023)
- [Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning](https://mlsanity.com/paper/2408.06259/) (2024)
- [The illusion of a perfect metric: Why evaluating AI's words is harder than it looks](https://mlsanity.com/paper/2508.13816/) (2025)
- [From Triage to Discharge: A Survey of NLP Tasks, Methods, and Open Challenges in the Emergency Department](https://mlsanity.com/paper/2608.23627/) (2026)
- [Preserving Privacy, Increasing Accessibility, and Reducing Cost: An On-Device Artificial Intelligence Model for Medical Transcription and Note Generation](https://mlsanity.com/paper/2507.03033/) (2025)

## Abstract (arXiv)

In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient's clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.