# Large Language Models as Medical Codes Selectors: a benchmark using the International Classification of Primary Care

arXiv: https://arxiv.org/abs/2507.14681 · HTML: https://mlsanity.com/paper/2507.14681/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 37 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Builds on (most cited first, 7 of 7)

- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [Beyond English: The Impact of Prompt Translation Strategies across Languages and Tasks in Multilingual LLMs](https://mlsanity.com/paper/2502.09331/) (2025)
- [Large language models are good medical coders, if provided with tools](https://mlsanity.com/paper/2407.12849/) (2024)
- [Review of Extreme Multilabel Classification](https://mlsanity.com/paper/2302.05971/) (2023)
- [ICXML: An In-Context Learning Framework for Zero-Shot Extreme Multi-Label Classification](https://mlsanity.com/paper/2311.09649/) (2023)
- [Zero-Shot ATC Coding with Large Language Models for Clinical Assessments](https://mlsanity.com/paper/2412.07743/) (2024)

## Abstract (arXiv)

Background: Medical coding structures healthcare data for research, quality monitoring, and policy. This study assesses the potential of large language models (LLMs) to assign ICPC-2 codes using the output of a domain-specific search engine. Methods: A dataset of 437 Brazilian Portuguese clinical expressions, each annotated with ICPC-2 codes, was used. A semantic search engine (OpenAI's text-embedding-3-large) retrieved candidates from 73,563 labeled concepts. Thirty-three LLMs were prompted with each query and retrieved results to select the best-matching ICPC-2 code. Performance was evaluated using F1-score, along with token usage, cost, response time, and format adherence. Results: Twenty-eight models achieved F1-score > 0.8; ten exceeded 0.85. Top performers included gpt-4.5-preview, o3, and gemini-2.5-pro. Retriever optimization can improve performance by up to 4 points. Most models returned valid codes in the expected format, with reduced hallucinations. Smaller models (<3B) struggled with formatting and input length. Conclusions: LLMs show strong potential for automating ICPC-2 coding, even without fine-tuning. This work offers a benchmark and highlights challenges, but findings are limited by dataset scope and setup. Broader, multilingual, end-to-end evaluations are needed for clinical validation.