# SafeRx-Agent: A Knowledge-Grounded Multi-Agent Framework for Safe and Explainable Medication Recommendation

arXiv: https://arxiv.org/abs/2605.29146 · HTML: https://mlsanity.com/paper/2605.29146/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 12 of 12)

- [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [MMedAgent: Learning to Use Medical Tools with Multi-modal Agent](https://mlsanity.com/paper/2407.02483/) (2024)
- [Med42-v2: A Suite of Clinical LLMs](https://mlsanity.com/paper/2408.06142/) (2024)
- [GAMENet: Graph Augmented MEmory Networks for Recommending Medication Combination](https://mlsanity.com/paper/1809.01852/) (2018)
- [UltraMedical: Building Specialized Generalists in Biomedicine](https://mlsanity.com/paper/2406.03949/) (2024)
- [SafeDrug: Dual Molecular Graph Encoders for Recommending Effective and Safe Drug Combinations](https://mlsanity.com/paper/2105.02711/) (2021)
- [Conditional Generation Net for Medication Recommendation](https://mlsanity.com/paper/2202.06588/) (2022)
- [Large Language Model Distilling Medication Recommendation Model](https://mlsanity.com/paper/2402.02803/) (2024)
- [Leave No Patient Behind: Enhancing Medication Recommendation for Rare Disease Patients](https://mlsanity.com/paper/2403.17745/) (2024)
- [Fine-grained List-wise Alignment for Generative Medication Recommendation](https://mlsanity.com/paper/2505.20218/) (2025)
- [Fine-grained Alignment of Large Language Models for General Medication Recommendation without Overprescription](https://mlsanity.com/paper/2503.03687/) (2025)

## Built on by (most cited first, 2 of 2)

- [HCOE: Hyperbolic Clinical Ontology Embeddings from Biomedical Language Models](https://mlsanity.com/paper/2609.30763/) (2026)
- [SMI: Ontology-Grounded Semantic IDs for Hierarchy-Aware Medical Concept Representation](https://mlsanity.com/paper/2610.18672/) (2026)

## Abstract (arXiv)

Medication recommendation predicts medications for patient visits, but existing methods still face two key challenges. At the model level, traditional drug recommendation methods only predict structured drug codes with limited evidence grounding, while LLM agents can use richer clinical context but may lack safety verification and traceability. At the task level, existing benchmarks often use broad medication categories, which ignore subgroup-level safety differences and can lead to risk overestimation. We introduce the first fine-grained medication recommendation setting based on fourth-level ATC code generation. We propose Safe Prescription Agent (SafeRx-Agent), a knowledge-grounded multi-agent framework that uses patient context, external clinical knowledge, and safety verification to recommend traceable medication sets. Experimental results on MIMIC-III and MIMIC-IV datasets show that SafeRx-Agent improves fine-grained medication prediction accuracy while controlling drug interactions, contraindications, and medication set size.