mlsanityData to 14 Oct 2026

RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU

arXiv 2504.15302 ↗First posted 17 Apr 2025cs.DC cs.OScited by 2

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) enhances large language model (LLM) generation quality by incorporating relevant external knowledge. However, deploying RAG on consumer-grade platforms is challenging due to limited memory and the increasing scale of both models and knowledge bases. In this work, we introduce RAGDoll, a resource-efficient, self-adaptive RAG serving system integrated with LLMs, specifically designed for resource-constrained platforms. RAGDoll exploits the insight that RAG retrieval and LLM generation impose different computational and memory demands, which in a traditional serial workflow result in substantial idle times and poor resource utilization. Based on this insight, RAGDoll decouples retrieval and generation into parallel pipelines, incorporating joint memory placement and dynamic batch scheduling strategies to optimize resource usage across diverse hardware devices and workloads. Extensive experiments demonstrate that RAGDoll adapts effectively to various hardware configurations and LLM scales, achieving up to 3.6 times speedup in average latency compared to serial RAG systems based on vLLM.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 33 of its 53 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

33 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving18 Mar
  2. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting11 Jul
  3. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design8 Mar
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  5. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models15 Oct
  6. 2022A Survey on Retrieval-Augmented Text Generation2 Feb

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 223
15 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  2. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 192
  3. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  4. 2024Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generationcited by 137
  5. 2018ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithmscited by 130
  6. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  7. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 91
  8. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  9. 2021FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Searchcited by 38
  10. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  11. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  12. 2024PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryptioncited by 10
  13. 2025Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batchingcited by 8
  14. 2025FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inferencecited by 4
  15. 2024Pareto-Optimized Open-Source LLMs for Healthcare via Context Retrievalcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  2. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
  2. 2026Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite
  3. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  4. 2026OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows
  5. 2025HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving
  6. 2025Harmonia: End-to-End RAG Serving Optimization
  7. 2024Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architectures
  8. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval