# Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations

arXiv: https://arxiv.org/abs/2311.13273 · HTML: https://mlsanity.com/paper/2311.13273/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 61 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 12 of 12)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [Better Summarization Evaluation with Word Embeddings for ROUGE](https://mlsanity.com/paper/1508.06034/) (2015)
- [Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation](https://mlsanity.com/paper/1706.09799/) (2017)
- [Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation](https://mlsanity.com/paper/2204.00447/) (2022)

## Built on by (most cited first, 1 of 1)

- [ClinBench-HPB: A Clinical Benchmark for Evaluating LLMs in Hepato-Pancreato-Biliary Diseases](https://mlsanity.com/paper/2506.00095/) (2025)

## Abstract (arXiv)

Generative Artificial Intelligence (AI) can be used to automatically generate medical reports based on transcripts of medical consultations. The aim is to reduce the administrative burden that healthcare professionals face. The accuracy of the generated reports needs to be established to ensure their correctness and usefulness. There are several metrics for measuring the accuracy of AI generated reports, but little work has been done towards the application of these metrics in medical reporting. A comparative experimentation of 10 accuracy metrics has been performed on AI generated medical reports against their corresponding General Practitioner's (GP) medical reports concerning Otitis consultations. The number of missing, incorrect, and additional statements of the generated reports have been correlated with the metric scores. In addition, we introduce and define a Composite Accuracy Score which produces a single score for comparing the metrics within the field of automated medical reporting. Findings show that based on the correlation study and the Composite Accuracy Score, the ROUGE-L and Word Mover's Distance metrics are the preferred metrics, which is not in line with previous work. These findings help determine the accuracy of an AI generated medical report, which aids the development of systems that generate medical reports for GPs to reduce the administrative burden.