# RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements

arXiv: https://arxiv.org/abs/2411.15700 · HTML: https://mlsanity.com/paper/2411.15700/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 23 of its 59 references that are in the map.
- First posted on arXiv 24 Nov 2024.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 23 of 23)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme](https://mlsanity.com/paper/1706.05075/) (2017)
- [LLMs Accelerate Annotation for Medical Information Extraction](https://mlsanity.com/paper/2312.02296/) (2023)
- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [Large Language Models for Disease Diagnosis: A Scoping Review](https://mlsanity.com/paper/2409.00097/) (2024)
- [BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers](https://mlsanity.com/paper/2404.18443/) (2024)
- [Pushing The Limit of LLM Capacity for Text Classification](https://mlsanity.com/paper/2402.07470/) (2024)
- [MT-Clinical BERT: Scaling Clinical Information Extraction with Multitask Learning](https://mlsanity.com/paper/2004.10220/) (2020)
- [Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction](https://mlsanity.com/paper/2401.13598/) (2024)
- [LLMs in Biomedicine: A study on clinical Named Entity Recognition](https://mlsanity.com/paper/2404.07376/) (2024)
- [Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness](https://mlsanity.com/paper/2405.08151/) (2024)
- [Deep Learning Approaches for Extracting Adverse Events and Indications of Dietary Supplements from Clinical Text](https://mlsanity.com/paper/2009.07780/) (2020)
- [A Conversational Agent System for Dietary Supplements Use](https://mlsanity.com/paper/2104.01543/) (2021)
- [Discovering novel drug-supplement interactions using a dietary supplements knowledge graph generated from the biomedical literature](https://mlsanity.com/paper/2106.12741/) (2021)

## Built on by (most cited first, 11 of 11)

- [MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learning](https://mlsanity.com/paper/2502.15954/) (2025)
- [Uncertainty-Aware Large Language Models for Explainable Disease Diagnosis](https://mlsanity.com/paper/2505.03467/) (2025)
- [Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Models](https://mlsanity.com/paper/2412.11455/) (2024)
- [An evaluation of DeepSeek Models in Biomedical Natural Language Processing](https://mlsanity.com/paper/2503.00624/) (2025)
- [Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation](https://mlsanity.com/paper/2509.04534/) (2025)
- [Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Study](https://mlsanity.com/paper/2604.00261/) (2026)
- [EPEE: Towards Efficient and Effective Foundation Models in Biomedicine](https://mlsanity.com/paper/2503.02053/) (2025)
- [Inference Gap in Domain Expertise and Machine Intelligence in Named Entity Recognition: Creation of and Insights from a Substance Use-related Dataset](https://mlsanity.com/paper/2508.19467/) (2025)
- [An Underexplored Frontier: Large Language Models for Rare Disease Patient Education and Communication -- A scoping review](https://mlsanity.com/paper/2604.14179/) (2026)
- [Retrieval-Augmented Large Language Models for Schema-Constrained Clinical Information Extraction](https://mlsanity.com/paper/2605.15467/) (2026)
- [ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models](https://mlsanity.com/paper/2606.03157/) (2026)

## Abstract (arXiv)

Objective: We aimed to develop an advanced multi-task large language model (LLM) framework to extract multiple types of information about dietary supplements (DS) from clinical records. Methods: We used four core DS information extraction tasks - namely, named entity recognition (NER: 2,949 clinical sentences), relation extraction (RE: 4,892 sentences), triple extraction (TE: 2,949 sentences), and usage classification (UC: 2,460 sentences) as our multitasks. We introduced a novel Retrieval-Augmented Multi-task Information Extraction (RAMIE) Framework, including: 1) employed instruction fine-tuning techniques with task-specific prompts, 2) trained LLMs for multiple tasks with improved storage efficiency and lower training costs, and 3) incorporated retrieval augmentation generation (RAG) techniques by retrieving similar examples from the training set. We compared RAMIE's performance to LLMs with instruction fine-tuning alone and conducted an ablation study to assess the contributions of multi-task learning and RAG to improved multitasking performance. Results: With the aid of the RAMIE framework, Llama2-13B achieved an F1 score of 87.39 (3.51% improvement) on the NER task and demonstrated outstanding performance on the RE task with an F1 score of 93.74 (1.15% improvement). For the TE task, Llama2-7B scored 79.45 (14.26% improvement), and MedAlpaca-7B achieved the highest F1 score of 93.45 (0.94% improvement) on the UC task. The ablation study revealed that while MTL increased efficiency with a slight trade-off in performance, RAG significantly boosted overall accuracy. Conclusion: This study presents a novel RAMIE framework that demonstrates substantial improvements in multi-task information extraction for DS-related data from clinical records. Our framework can potentially be applied to other domains.