# SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment

arXiv: https://arxiv.org/abs/2605.04012 · HTML: https://mlsanity.com/paper/2605.04012/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 8 of its 45 references that are in the map.
- First posted on arXiv 5 May 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 8 of 8)

- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks](https://mlsanity.com/paper/2505.23802/) (2025)
- [Advancing Conversational Diagnostic AI with Multimodal Reasoning](https://mlsanity.com/paper/2505.04653/) (2025)
- [Towards Conversational AI for Disease Management](https://mlsanity.com/paper/2503.06074/) (2025)
- [Towards physician-centered oversight of conversational diagnostic AI](https://mlsanity.com/paper/2507.15743/) (2025)
- [Toward the Autonomous AI Doctor: Quantitative Benchmarking of an Autonomous Agentic AI Versus Board-Certified Clinicians in a Real World Setting](https://mlsanity.com/paper/2507.22902/) (2025)

## Built on by (most cited first, 4 of 4)

- [Towards a General Intelligence and Interface for Wearable Health Data](https://mlsanity.com/paper/2605.22759/) (2026)
- [ResidencyRL: Reinforcement Learning in Simulated Clinical Environments](https://mlsanity.com/paper/2608.07418/) (2026)
- [Evaluating the Utility of Personal Health Records in Personalized Health AI](https://mlsanity.com/paper/2605.18937/) (2026)
- [Reasoning in Real World Clinical Care: Why Large Language Models Are Not Yet Safe for Autonomous Clinical Decision Support](https://mlsanity.com/paper/2607.28677/) (2026)

## Abstract (arXiv)

Language models excel at diagnostic assessments on curated medical case-studies and vignettes, performing on par with, or better than, clinical professionals. However, existing studies focus on complex scenarios with rich context making it difficult to draw conclusions about how these systems perform for patients reporting symptoms in everyday life. We deployed SymptomAI, a set of conversational AI agents for end-to-end patient interviewing and differential diagnosis (DDx), via the Fitbit app in a study that randomized participants (N=13,917) to interact with five AI agents. This corpus captures diverse communication and a realistic distribution of illnesses from a real world population. A subset of 1,228 participants reported a clinician-provided diagnosis, and 517 of these were further evaluated by a panel of clinicians during over 250 hours of annotation. SymptomAI DDx were significantly more accurate (OR = 2.56, p < 0.001) than those from independent clinicians given the same dialogue in a blinded randomized comparison. Moreover, agentic strategies which conduct a dedicated symptom interview that elicit additional symptom information before providing a diagnosis, perform substantially better than baseline, user-guided conversations (p < 0.001). An auxiliary analysis on 1,509 conversations from a general US population panel validated that these results generalize beyond wearable device users. We used SymptomAI diagnoses as labels for all 13,917 participants to analyze over 500,000 days of wearable metrics across nearly 400 unique conditions. We identified strong associations between acute infections and physiological shifts (e.g., OR > 7 for influenza). While limited by self-reported ground truth, these results demonstrate the benefits of a dedicated and complete symptom interview compared to a user-guided symptom discussion, which is the default of most consumer LLMs.