# Automatic Construction of Clinical Scoring Systems with LLM Agents

arXiv: https://arxiv.org/abs/2601.22324 · HTML: https://mlsanity.com/paper/2601.22324/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 103 references that are in the map.
- First posted on arXiv 29 Jan 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 21 of 21)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model](https://mlsanity.com/paper/1511.01644/) (2015)
- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [Ethical Machine Learning in Health Care](https://mlsanity.com/paper/2009.10576/) (2020)
- [Optimal Sparse Decision Trees](https://mlsanity.com/paper/1904.12847/) (2019)
- [Supersparse Linear Integer Models for Optimized Medical Scoring Systems](https://mlsanity.com/paper/1502.04269/) (2015)
- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [Learning Optimized Risk Scores](https://mlsanity.com/paper/1610.00168/) (2016)
- [The Voice of Optimization](https://mlsanity.com/paper/1812.09991/) (2018)
- [FasterRisk: Fast and Accurate Interpretable Risk Scores](https://mlsanity.com/paper/2210.05846/) (2022)
- [LLM-based feature generation from text for interpretable machine learning](https://mlsanity.com/paper/2409.07132/) (2024)
- [Learning Optimal Predictive Checklists](https://mlsanity.com/paper/2112.01020/) (2021)
- [Decision Tree Induction Through LLMs via Semantically-Aware Evolution](https://mlsanity.com/paper/2503.14217/) (2025)

## Built on by (most cited first, 1 of 1)

- [Tracing the Heart: An Evidence-Linked Pipeline for Heart-Failure Feature Engineering](https://mlsanity.com/paper/2608.06366/) (2026)

## Abstract (arXiv)

Modern clinical practice relies on evidence-based guidelines implemented as compact scoring systems composed of a small number of interpretable decision rules. While machine-learning models achieve strong performance, many fail to translate into routine clinical use due to misalignment with workflow constraints such as memorability, auditability, and bedside execution. We argue that this gap arises not from insufficient predictive power, but from optimizing over model classes that are incompatible with guideline deployment. Deployable guidelines often take the form of unit-weighted clinical checklists, formed by thresholding the sum of binary rules, but learning such scores requires searching an exponentially large discrete space of possible rule sets. We introduce AgentScore, which performs semantically guided optimization in this space by using LLMs to propose candidate rules and a deterministic, data-grounded verification-and-selection loop to enforce statistical validity and deployability constraints. Across eight clinical prediction tasks, AgentScore outperforms existing score-generation methods and achieves AUROC comparable to more flexible interpretable models despite operating under stronger structural constraints. On two additional externally validated tasks, AgentScore achieves higher discrimination than established guideline-based scores.