# Large Multimodal Model based Standardisation of Pathology Reports with Confidence and their Prognostic Significance

arXiv: https://arxiv.org/abs/2405.02040 · HTML: https://mlsanity.com/paper/2405.02040/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 11 of 11)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Large Language Models Understand and Can be Enhanced by Emotional Stimuli](https://mlsanity.com/paper/2307.11760/) (2023)
- [Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation](https://mlsanity.com/paper/2310.16809/) (2023)
- [BioIE: Biomedical Information Extraction with Multi-head Attention Enhanced Graph Convolutional Network](https://mlsanity.com/paper/2110.13683/) (2021)

## Built on by (most cited first, 2 of 2)

- [MedPAO: A Protocol-Driven Agent for Structuring Medical Reports](https://mlsanity.com/paper/2510.04623/) (2025)
- [IHC-LLMiner: Automated extraction of tumour immunohistochemical profiles from PubMed abstracts using large language models](https://mlsanity.com/paper/2504.00748/) (2025)

## Abstract (arXiv)

Pathology reports are rich in clinical and pathological details but are often presented in free-text format. The unstructured nature of these reports presents a significant challenge limiting the accessibility of their content. In this work, we present a practical approach based on the use of large multimodal models (LMMs) for automatically extracting information from scanned images of pathology reports with the goal of generating a standardised report specifying the value of different fields along with estimated confidence about the accuracy of the extracted fields. The proposed approach overcomes limitations of existing methods which do not assign confidence scores to extracted fields limiting their practical use. The proposed framework uses two stages of prompting a Large Multimodal Model (LMM) for information extraction and validation. The framework generalises to textual reports from multiple medical centres as well as scanned images of legacy pathology reports. We show that the estimated confidence is an effective indicator of the accuracy of the extracted information that can be used to select only accurately extracted fields. We also show the prognostic significance of structured and unstructured data from pathology reports and show that the automatically extracted field values significant prognostic value for patient stratification. The framework is available for evaluation via the URL: https://labieb.dcs.warwick.ac.uk/.