# Teaching agentic AI to generalize expert diagnostic reasoning in rare diseases

arXiv: https://arxiv.org/abs/2606.16149 · HTML: https://mlsanity.com/paper/2606.16149/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 15 Jun 2026.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)
- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning](https://mlsanity.com/paper/2507.19457/) (2025)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [Inference-Time Scaling for Generalist Reward Modeling](https://mlsanity.com/paper/2504.02495/) (2025)
- [On-Policy Context Distillation for Language Models](https://mlsanity.com/paper/2602.12275/) (2026)
- [MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making](https://mlsanity.com/paper/2404.15155/) (2024)
- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings](https://mlsanity.com/paper/2604.04323/) (2026)
- [Doc-to-LoRA: Learning to Instantly Internalize Contexts](https://mlsanity.com/paper/2602.15902/) (2026)
- [Reward Is Enough: LLMs Are In-Context Reinforcement Learners](https://mlsanity.com/paper/2506.06303/) (2025)
- [End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoning](https://mlsanity.com/paper/2508.15746/) (2025)
- [A Specialized Large Language Model for Clinical Reasoning and Diagnosis in Rare Diseases](https://mlsanity.com/paper/2511.14638/) (2025)

## Built on by (most cited first, 1 of 1)

- [Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning](https://mlsanity.com/paper/2608.16831/) (2026)

## Abstract (arXiv)

Rare disease diagnosis depends on expert reasoning that is scarce and difficult to transfer. Large language models rank the correct disease first in only 35.4% of benchmark cases and often rely on learned phenotype-disease associations rather than reusable diagnostic reasoning strategies. We developed liteOdyssey through Policy Iteration with Human Feedback, a process in which model failures and expert corrections are iteratively consolidated into a clinician-gated, natural-language policy executed by a language model. Across 1,243 public benchmark cases spanning 722 rare diseases, liteOdyssey ranked the correct disease first in 59.3% of cases versus 26.5% without the policy, with comparable gains in cases involving diseases excluded from policy development. The same policy transferred across model families and sizes without retraining. Adaptation of the policy to the Undiagnosed Diseases Network (UDN) improved diagnostic accuracy among 515 UDN patients, with gains confirmed by blinded physician adjudication. These results show that expert reasoning can be externalized into an inspectable and revisable natural-language policy that generalizes across rare diseases, transfers across model backbones, and adapts to a real-world patient cohort.