# Evaluating Hierarchical Clinical Document Classification Using Reasoning-Based LLMs

arXiv: https://arxiv.org/abs/2507.03001 · HTML: https://mlsanity.com/paper/2507.03001/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 11 of 11)

- [Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning](https://mlsanity.com/paper/2310.01061/) (2023)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [Reasoning Models Can Be Effective Without Thinking](https://mlsanity.com/paper/2504.09858/) (2025)
- [Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA](https://mlsanity.com/paper/2406.17419/) (2024)
- [Do We Still Need Clinical Language Models?](https://mlsanity.com/paper/2302.08091/) (2023)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems](https://mlsanity.com/paper/2407.10701/) (2024)
- [Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases](https://mlsanity.com/paper/2503.04691/) (2025)
- [Natural Language Processing in Electronic Health Records in Relation to Healthcare Decision-making: A Systematic Review](https://mlsanity.com/paper/2306.12834/) (2023)
- [Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration](https://mlsanity.com/paper/2405.16546/) (2024)
- [Can Reasoning LLMs Enhance Clinical Document Classification?](https://mlsanity.com/paper/2504.08040/) (2025)

## Built on by (most cited first, 2 of 2)

- [MedDCR: Learning to Design Agentic Workflows for Medical Coding](https://mlsanity.com/paper/2511.13361/) (2025)
- [KREL: Automatic Medical Coding via Knowledge-Guided Reasoning over Clinical Evidence with LLMs](https://mlsanity.com/paper/2608.20887/) (2026)

## Abstract (arXiv)

This study evaluates how well large language models (LLMs) can classify ICD-10 codes from hospital discharge summaries, a critical but error-prone task in healthcare. Using 1,500 summaries from the MIMIC-IV dataset and focusing on the 10 most frequent ICD-10 codes, the study tested 11 LLMs, including models with and without structured reasoning capabilities. Medical terms were extracted using a clinical NLP tool (cTAKES), and models were prompted in a consistent, coder-like format. None of the models achieved an F1 score above 57%, with performance dropping as code specificity increased. Reasoning-based models generally outperformed non-reasoning ones, with Gemini 2.5 Pro performing best overall. Some codes, such as those related to chronic heart disease, were classified more accurately than others. The findings suggest that while LLMs can assist human coders, they are not yet reliable enough for full automation. Future work should explore hybrid methods, domain-specific model training, and the use of structured clinical data.