# MALADE: Orchestration of LLM-powered Agents with Retrieval Augmented Generation for Pharmacovigilance arXiv: https://arxiv.org/abs/2408.01869 ยท HTML: https://mlsanity.com/paper/2408.01869/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. - Builds on 18 of its 81 references that are in the map. - First posted on arXiv 3 Aug 2024. ## Placed in - Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/) - LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/) ## Builds on (most cited first, 18 of 18) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013) - [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023) - [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023) - [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023) - [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022) - [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023) - [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023) - [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023) - [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023) - [MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning](https://mlsanity.com/paper/2311.10537/) (2023) - [Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study](https://mlsanity.com/paper/2402.15663/) (2024) - [DAEDRA: A language model for predicting outcomes in passive pharmacovigilance reporting](https://mlsanity.com/paper/2402.10951/) (2024) ## Built on by (most cited first, 11 of 11) - [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025) - [A Survey on the Optimization of Large Language Model-based Agents](https://mlsanity.com/paper/2503.12434/) (2025) - [Formal Policy Enforcement for Real-World Agentic Systems](https://mlsanity.com/paper/2602.16708/) (2026) - [MAP: Multi-user Personalization with Collaborative LLM-powered Agents](https://mlsanity.com/paper/2503.12757/) (2025) - [Experience as a Compass: Multi-agent RAG with Evolving Orchestration and Agent Prompts](https://mlsanity.com/paper/2604.00901/) (2026) - [The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy](https://mlsanity.com/paper/2607.11175/) (2026) - [Retrieval Augmented Large Language Model System for Comprehensive Drug Contraindications](https://mlsanity.com/paper/2508.06145/) (2025) - [Zero-shot reasoning for simulating scholarly peer-review](https://mlsanity.com/paper/2510.02027/) (2025) - [SciHorizon-DataEVA: An Agentic System for AI-Readiness Evaluation of Heterogeneous Scientific Data](https://mlsanity.com/paper/2604.26645/) (2026) - [Agentic AI in medicine: architectures, applications, evaluation, and challenges for clinical translation](https://mlsanity.com/paper/2607.25489/) (2026) - [Evaluating Agentic Bioinformatics through Function, Evidence, and Validation](https://mlsanity.com/paper/2607.27556/) (2026) ## Abstract (arXiv) In the era of Large Language Models (LLMs), given their remarkable text understanding and generation abilities, there is an unprecedented opportunity to develop new, LLM-based methods for trustworthy medical knowledge synthesis, extraction and summarization. This paper focuses on the problem of Pharmacovigilance (PhV), where the significance and challenges lie in identifying Adverse Drug Events (ADEs) from diverse text sources, such as medical literature, clinical notes, and drug labels. Unfortunately, this task is hindered by factors including variations in the terminologies of drugs and outcomes, and ADE descriptions often being buried in large amounts of narrative text. We present MALADE, the first effective collaborative multi-agent system powered by LLM with Retrieval Augmented Generation for ADE extraction from drug label data. This technique involves augmenting a query to an LLM with relevant information extracted from text resources, and instructing the LLM to compose a response consistent with the augmented data. MALADE is a general LLM-agnostic architecture, and its unique capabilities are: (1) leveraging a variety of external sources, such as medical literature, drug labels, and FDA tools (e.g., OpenFDA drug information API), (2) extracting drug-outcome association in a structured format along with the strength of the association, and (3) providing explanations for established associations. Instantiated with GPT-4 Turbo or GPT-4o, and FDA drug label data, MALADE demonstrates its efficacy with an Area Under ROC Curve of 0.90 against the OMOP Ground Truth table of ADEs. Our implementation leverages the Langroid multi-agent LLM framework and can be found at https://github.com/jihyechoi77/malade.