mlsanityData to 14 Oct 2026

Efficient Distributed Retrieval-Augmented Generation for Enhancing Language Model Performance

arXiv 2504.11197 ↗First posted 15 Apr 2025cs.LG cs.AI cs.DCcited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Small language models (SLMs) support efficient deployments on resource-constrained edge devices, but their limited capacity compromises inference performance. Retrieval-augmented generation (RAG) is a promising solution to enhance model performance by integrating external databases, without requiring intensive on-device model retraining. However, large-scale public databases and user-specific private contextual documents are typically located on the cloud and the device separately, while existing RAG implementations are primarily centralized. To bridge this gap, we propose DRAGON, a distributed RAG framework to enhance on-device SLMs through both general and personal knowledge without the risk of leaking document privacy. Specifically, DRAGON decomposes multi-document RAG into multiple parallel token generation processes performed independently and locally on the cloud and the device, and employs a newly designed Speculative Aggregation, a dual-side speculative algorithm to avoid frequent output synchronization between the cloud and device. A new scheduling algorithm is further introduced to identify the optimal aggregation side based on real-time network conditions. Evaluations on real-world hardware testbed demonstrate a significant performance improvement of DRAGON-up to 1.9x greater gains over standalone SLM compared to the centralized RAG, substantial reduction in per-token latency, and negligible Time to First Token (TTFT) overhead.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 58 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

33 of 58 references.

References · 33

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2017Billion-scale similarity search with GPUscited by 2,497
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
21 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  2. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  4. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  5. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  7. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  8. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  9. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  10. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  11. 2020SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloudcited by 112
  12. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  13. 2022Parallel Context Windows for Large Language Modelscited by 81
  14. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  15. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 56
  16. 2017Mobile Edge Computing: A Survey on Architecture and Computation Offloadingcited by 53
  17. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  18. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  19. 2021Auto-Split: A General Framework of Collaborative Edge-Cloud AIcited by 30
  20. 2024Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inferencecited by 7
  21. 2023Hybrid-RACA: Hybrid Retrieval-Augmented Composition Assistance for Real-time Text Predictioncited by 4

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025DGRAG: Distributed Graph-based Retrieval-Augmented Generation in Edge-Cloud Systemscited by 0
  2. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Cloud to Crowd: Democratizing LLM Service with Decentralized Edge Collaboration for RAG ↗
  2. 2026CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolation
  3. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  4. 2024Accelerating Retrieval-Augmented Generation
  5. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  6. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processing
  7. 2024EACO-RAG: Towards Distributed Tiered LLM Deployment using Edge-Assisted and Collaborative RAG with Adaptive Knowledge Update
  8. 2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs