# HyFedRAG: A Federated Retrieval-Augmented Generation Framework for Heterogeneous and Privacy-Sensitive Data

arXiv: https://arxiv.org/abs/2509.06444 · HTML: https://mlsanity.com/paper/2509.06444/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [Dense Text Retrieval based on Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.14876/) (2022)
- [Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval](https://mlsanity.com/paper/2312.15503/) (2023)
- [Open Domain Question Answering over Tables via Dense Retrieval](https://mlsanity.com/paper/2103.12011/) (2021)
- [TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption](https://mlsanity.com/paper/2104.03152/) (2021)
- [Open Domain Question Answering with A Unified Knowledge Interface](https://mlsanity.com/paper/2110.08417/) (2021)
- [Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering](https://mlsanity.com/paper/2108.02866/) (2021)
- [OpenTab: Advancing Large Language Models as Open-domain Table Reasoners](https://mlsanity.com/paper/2402.14361/) (2024)
- [FedCache: A Knowledge Cache-driven Federated Learning Architecture for Personalized Edge Intelligence](https://mlsanity.com/paper/2308.07816/) (2023)
- [Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding](https://mlsanity.com/paper/2305.14232/) (2023)
- [Question Decomposition Tree for Answering Complex Questions over Knowledge Bases](https://mlsanity.com/paper/2306.07597/) (2023)
- [Accelerating Deep Learning Inference via Learned Caches](https://mlsanity.com/paper/2101.07344/) (2021)
- [RAG-based Question Answering over Heterogeneous Data and Text](https://mlsanity.com/paper/2412.07420/) (2024)

## Built on by (most cited first, 4 of 4)

- [Stable-RAG: Mitigating Retrieval-Permutation-Induced Hallucinations in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.02993/) (2026)
- [Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions](https://mlsanity.com/paper/2511.11347/) (2025)
- [Privacy-Preserving Text Sanitization for Distributed Agents Collaboration via Disentangled Representations](https://mlsanity.com/paper/2606.15335/) (2026)
- [As We May Search](https://mlsanity.com/paper/2606.29652/) (2026)

## Abstract (arXiv)

Centralized RAG pipelines struggle with heterogeneous and privacy-sensitive data, especially in distributed healthcare settings where patient data spans SQL, knowledge graphs, and clinical notes. Clinicians face difficulties retrieving rare disease cases due to privacy constraints and the limitations of traditional cloud-based RAG systems in handling diverse formats and edge devices. To address this, we introduce HyFedRAG, a unified and efficient Federated RAG framework tailored for Hybrid data modalities. By leveraging an edge-cloud collaborative mechanism, HyFedRAG enables RAG to operate across diverse data sources while preserving data privacy. Our key contributions are: (1) We design an edge-cloud collaborative RAG framework built on Flower, which supports querying structured SQL data, semi-structured knowledge graphs, and unstructured documents. The edge-side LLMs convert diverse data into standardized privacy-preserving representations, and the server-side LLMs integrates them for global reasoning and generation. (2) We integrate lightweight local retrievers with privacy-aware LLMs and provide three anonymization tools that enable each client to produce semantically rich, de-identified summaries for global inference across devices. (3) To optimize response latency and reduce redundant computation, we design a three-tier caching strategy consisting of local cache, intermediate representation cache, and cloud inference cache. Experimental results on PMC-Patients demonstrate that HyFedRAG outperforms existing baselines in terms of retrieval quality, generation consistency, and system efficiency. Our framework offers a scalable and privacy-compliant solution for RAG over structural-heterogeneous data, unlocking the potential of LLMs in sensitive and diverse data environments.