# Human-AI Co-design for Clinical Prediction Models

arXiv: https://arxiv.org/abs/2601.09072 · HTML: https://mlsanity.com/paper/2601.09072/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 69 references that are in the map.
- First posted on arXiv 14 Jan 2026.

## Builds on (most cited first, 24 of 24)

- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records](https://mlsanity.com/paper/2203.03540/) (2022)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [Interactive Concept Bottleneck Models](https://mlsanity.com/paper/2212.07430/) (2022)
- [Supersparse Linear Integer Models for Optimized Medical Scoring Systems](https://mlsanity.com/paper/1502.04269/) (2015)
- [Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces](https://mlsanity.com/paper/2309.14459/) (2023)
- [Scalable Bayesian Rule Lists](https://mlsanity.com/paper/1602.08610/) (2016)
- [Large Language Models to Identify Social Determinants of Health in Electronic Health Records](https://mlsanity.com/paper/2308.06354/) (2023)
- [Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability](https://mlsanity.com/paper/2207.09615/) (2022)
- [A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration](https://mlsanity.com/paper/2404.00405/) (2024)
- [Learning Interpretable Style Embeddings via Prompting LLMs](https://mlsanity.com/paper/2305.12696/) (2023)
- [Tree Prompting: Efficient Task Adaptation without Fine-Tuning](https://mlsanity.com/paper/2310.14034/) (2023)
- [Learning Interpretable Concept-Based Models with Human Feedback](https://mlsanity.com/paper/2012.02898/) (2020)
- [Integrating Clinical Knowledge into Concept Bottleneck Models](https://mlsanity.com/paper/2407.06600/) (2024)
- [Bayesian Concept Bottleneck Models with LLM Priors](https://mlsanity.com/paper/2410.15555/) (2024)
- [CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models](https://mlsanity.com/paper/2302.12343/) (2023)
- [A General Framework for Producing Interpretable Semantic Text Embeddings](https://mlsanity.com/paper/2410.03435/) (2024)
- [Tree-Based Leakage Inspection and Control in Concept Bottleneck Models](https://mlsanity.com/paper/2410.06352/) (2024)
- [Constructing Concept-based Models to Mitigate Spurious Correlations with Minimal Human Effort](https://mlsanity.com/paper/2407.08947/) (2024)
- [Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks](https://mlsanity.com/paper/2502.10526/) (2025)
- [ACT: Agentic Classification Tree](https://mlsanity.com/paper/2509.26433/) (2025)

## Built on by (most cited first, 9 of 9)

- [Test-time Recursive Thinking: Self-Improvement without External Feedback](https://mlsanity.com/paper/2602.03094/) (2026)
- [Do explanations generalize across large reasoning models?](https://mlsanity.com/paper/2601.11517/) (2026)
- [Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries](https://mlsanity.com/paper/2606.28960/) (2026)
- [Sanity Checks for Agentic Data Science](https://mlsanity.com/paper/2604.11003/) (2026)
- [Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks](https://mlsanity.com/paper/2602.22719/) (2026)
- [Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models](https://mlsanity.com/paper/2604.13332/) (2026)
- [Agentic-imodels: Evolving agentic interpretability tools via autoresearch](https://mlsanity.com/paper/2605.03808/) (2026)
- [From Fuzzy to Formal: Scaling Hospital Quality Improvement with AI](https://mlsanity.com/paper/2604.20055/) (2026)
- [Data-Efficient Adaptation of LLMs via Attention Head Reweighting](https://mlsanity.com/paper/2607.13425/) (2026)

## Abstract (arXiv)

Developing safe, effective, and practically useful clinical prediction models (CPMs) traditionally requires iterative collaboration between clinical experts, data scientists, and informaticists. This process refines the often small but critical details of the model building process, such as which features/patients to include and how clinical categories should be defined. However, this traditional collaboration process is extremely time- and resource-intensive, resulting in only a small fraction of CPMs reaching clinical practice. This challenge intensifies when teams attempt to incorporate unstructured clinical notes, which can contain an enormous number of concepts. To address this challenge, we introduce HACHI, an iterative human-in-the-loop framework that uses AI agents to accelerate the development of fully interpretable CPMs by enabling the exploration of concepts in clinical notes. HACHI alternates between (i) an AI agent rapidly exploring and evaluating candidate concepts in clinical notes and (ii) clinical and domain experts providing feedback to improve the CPM learning process. HACHI defines concepts as simple yes-no questions that are used in linear models, allowing the clinical AI team to transparently review, refine, and validate the CPM learned in each round. In two real-world prediction tasks (acute kidney injury and traumatic brain injury), HACHI outperforms existing approaches, surfaces new clinically relevant concepts not included in commonly-used CPMs, and improves model generalizability across clinical sites and time periods. Furthermore, HACHI reveals the critical role of the clinical AI team, such as directing the AI agent to explore concepts that it had not previously considered, adjusting the granularity of concepts it considers, changing the objective function to better align with the clinical objectives, and identifying issues of data bias and leakage.