Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care
Not on the map. Why?
Abstract (arXiv)
Specialist epilepsy expertise is scarce in resource-constrained settings, making LLM-based decision support attractive for frontline clinicians managing longitudinal treatment. Such systems must adapt to local prescribing practice and know when to defer. We study this problem in Ugandan pediatric epilepsy care, predicting anti-seizure medication regimens from longitudinal unstructured clinic notes. Standard prompting achieves non-trivial agreement with physician prescriptions, but neurologist review shows that many errors reflect distribution-miscalibrated prescribing defaults rather than failures to parse the local record. We introduce MANANA, a non-parametric prompt-learning framework that learns local prescribing guidance from a small patient-level training set. MANANA converts observed prescription errors into auditable prompt memories, instantiated in single-agent and multi-agent variants, and improves over classical ML models, direct LLM prompting, and prompt-optimization baselines across two independently collected Ugandan cohorts. We further propose Bayesian prompt averaging, which converts the learned prompt trajectory into prescription likelihoods and an uncertainty-based deferral signal. On the independently collected held-out cohort, this improves visit-level top-3 prescription accuracy by 4-8 percentage points over prompt-optimization baselines and enables selective prediction: the system can auto-handle the most confident half of cases at 95% precision, or the most confident quarter at 99% precision, while deferring lower-confidence cases for specialist review.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 35 of its 63 references that are in the map. First posted on arXiv 30 Jun 2026.
Builds on
35 of 63 references.
References · 35
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2022Large Language Models Encode Clinical Knowledgecited by 1,830
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
23 more
- 2023Large Language Models as Optimizerscited by 864
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
- 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 542
- 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
- 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
- 2024TextGrad: Automatic "Differentiation" via Textcited by 273
- 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
- 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
- 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
- 2018GAMENet: Graph Augmented MEmory Networks for Recommending Medication Combinationcited by 83
- 2024HSEvo: Elevating Automatic Heuristic Design with Diversity-Driven Harmony Search and Genetic Algorithm Using LLMscited by 73
- 2025MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentscited by 62
- 2018Direct Uncertainty Prediction for Medical Second Opinionscited by 61
- 2021SafeDrug: Dual Molecular Graph Encoders for Recommending Effective and Safe Drug Combinationscited by 54
- 2021Change Matters: Medication Change Prediction with Recurrent Residual Networkscited by 38
- 2024RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatmentcited by 6
- 2024Large Language Models for Medical Forecasting -- Foresight 2cited by 5
- 2025Fine-grained List-wise Alignment for Generative Medication Recommendationcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.