# Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach arXiv: https://arxiv.org/abs/2407.00978 ยท HTML: https://mlsanity.com/paper/2407.00978/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 25 of its 46 references that are in the map. - First posted on arXiv 1 Jul 2024. ## Builds on (most cited first, 25 of 25) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023) - [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015) - [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023) - [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022) - [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023) - [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019) - [Evolution of Semantic Similarity -- A Survey](https://mlsanity.com/paper/2004.13820/) (2020) - [Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow](https://mlsanity.com/paper/2306.07209/) (2023) - [Interactive AI with Retrieval-Augmented Generation for Next Generation Networking](https://mlsanity.com/paper/2401.11391/) (2024) - [Diffusion-based Reinforcement Learning for Edge-enabled AI-Generated Content Services](https://mlsanity.com/paper/2303.13052/) (2023) - [RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training](https://mlsanity.com/paper/2303.00534/) (2023) - [Large Language Models for Networking: Applications, Enabling Techniques, and Challenges](https://mlsanity.com/paper/2311.17474/) (2023) - [AI-Generated Incentive Mechanism and Full-Duplex Semantic Communications for Information Sharing](https://mlsanity.com/paper/2303.01896/) (2023) - [From Generative AI to Generative Internet of Things: Fundamentals, Framework, and Outlooks](https://mlsanity.com/paper/2310.18382/) (2023) - [Blockchain-empowered Federated Learning for Healthcare Metaverses: User-centric Incentive Mechanism with Optimal Data Freshness](https://mlsanity.com/paper/2307.15975/) (2023) - [Cached Model-as-a-Resource: Provisioning Large Language Model Agents for Edge Intelligence in Space-air-ground Integrated Networks](https://mlsanity.com/paper/2403.05826/) (2024) - [Filling the Missing: Exploring Generative AI for Enhanced Federated Learning over Heterogeneous Mobile Edge Devices](https://mlsanity.com/paper/2310.13981/) (2023) - [Generative AI-Driven Human Digital Twin in IoT-Healthcare: A Comprehensive Survey](https://mlsanity.com/paper/2401.13699/) (2024) - [Diffusion Model-based Incentive Mechanism with Prospect Theory for Edge AIGC Services in 6G IoT](https://mlsanity.com/paper/2407.10979/) (2024) - [Blockchain-assisted Twin Migration for Vehicular Metaverses: A Game Theory Approach](https://mlsanity.com/paper/2309.02173/) (2023) - [Optimizing Mobile-Edge AI-Generated Everything (AIGX) Services by Prompt Engineering: Fundamental, Framework, and Case Study](https://mlsanity.com/paper/2309.01065/) (2023) - [Digital Twin-Empowered Task Assignment in Aerial MEC Network: A Resource Coalition Cooperation Approach with Generative Model](https://mlsanity.com/paper/2405.01555/) (2024) - [Privacy-preserving Pseudonym Schemes for Personalized 3D Avatars in Mobile Social Metaverses](https://mlsanity.com/paper/2406.11208/) (2024) ## Built on by (most cited first, 7 of 7) - [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025) - [Diffusion Models for Future Networks and Communications: A Comprehensive Survey](https://mlsanity.com/paper/2508.01586/) (2025) - [Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments](https://mlsanity.com/paper/2505.01307/) (2025) - [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025) - [An Agentic AI Framework with Large Language Models and Chain-of-Thought for UAV-Assisted Logistics Scheduling with Mobile Edge Computing](https://mlsanity.com/paper/2605.13221/) (2026) - [MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning](https://mlsanity.com/paper/2510.21093/) (2025) - [Exqutor: Extended Query Optimizer for Vector-augmented Analytical Queries](https://mlsanity.com/paper/2512.09695/) (2025) ## Abstract (arXiv) Secure data management and effective data sharing have become paramount in the rapidly evolving healthcare landscape, especially with the growing integration of the Internet of Medical Things (IoMT). The rise of generative artificial intelligence has further elevated Multi-modal Large Language Models (MLLMs) as essential tools for managing and optimizing healthcare data in IoMT. MLLMs can support multi-modal inputs and generate diverse types of content by leveraging large-scale training on vast amounts of multi-modal data. However, critical challenges persist in developing medical MLLMs, including security and freshness issues of healthcare data, affecting the output quality of MLLMs. To this end, in this paper, we propose a hybrid Retrieval-Augmented Generation (RAG)-empowered medical MLLM framework for healthcare data management. This framework leverages a hierarchical cross-chain architecture to facilitate secure data training. Moreover, it enhances the output quality of MLLMs through hybrid RAG, which employs multi-modal metrics to filter various unimodal RAG results and incorporates these retrieval results as additional inputs to MLLMs. Additionally, we employ age of information to indirectly evaluate the data freshness impact of MLLMs and utilize contract theory to incentivize healthcare data holders to share their fresh data, mitigating information asymmetry during data sharing. Finally, we utilize a generative diffusion model-based deep reinforcement learning algorithm to identify the optimal contract for efficient data sharing. Numerical results demonstrate the effectiveness of the proposed schemes, which achieve secure and efficient healthcare data management.