# Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study

arXiv: https://arxiv.org/abs/2510.12813 · HTML: https://mlsanity.com/paper/2510.12813/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 14 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 4 of 4)

- [Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health](https://mlsanity.com/paper/2306.10070/) (2023)
- [A Comprehensive Overview of Backdoor Attacks in Large Language Models within Communication Networks](https://mlsanity.com/paper/2308.14367/) (2023)
- [Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification](https://mlsanity.com/paper/2304.02496/) (2023)
- [Deep learning-based NLP Data Pipeline for EHR Scanned Document Information Extraction](https://mlsanity.com/paper/2110.11864/) (2021)

## Abstract (arXiv)

Electronic health records contain inconsistently structured or free-text data, requiring efficient preprocessing to enable predictive health care models. Although artificial intelligence-driven natural language processing tools show promise for automating diagnosis classification, their comparative performance and clinical reliability require systematic evaluation. The aim of this study is to evaluate the performance of 4 large language models (GPT-3.5, GPT-4o, Llama 3.2, and Gemini 1.5) and BioBERT in classifying cancer diagnoses from structured and unstructured electronic health records data. We analyzed 762 unique diagnoses (326 International Classification of Diseases (ICD) code descriptions, 436free-text entries) from 3456 records of patients with cancer. Models were tested on their ability to categorize diagnoses into 14predefined categories. Two oncology experts validated classifications. BioBERT achieved the highest weighted macro F1-score for ICD codes (84.2) and matched GPT-4o in ICD code accuracy (90.8). For free-text diagnoses, GPT-4o outperformed BioBERT in weighted macro F1-score (71.8 vs 61.5) and achieved slightly higher accuracy (81.9 vs 81.6). GPT-3.5, Gemini, and Llama showed lower overall performance on both formats. Common misclassification patterns included confusion between metastasis and central nervous system tumors, as well as errors involving ambiguous or overlapping clinical terminology. Although current performance levels appear sufficient for administrative and research use, reliable clinical applications will require standardized documentation practices alongside robust human oversight for high-stakes decision-making.