mlsanityData to 14 Oct 2026

Harmonia: End-to-End RAG Serving Optimization

arXiv 2505.07833 ↗First posted 1 May 2025cs.DC cs.AI cs.MAcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) improves the reliability of large language models by integrating external knowledge, but serving RAG pipelines efficiently is challenging because requests traverse heterogeneous components spanning LLM inference, databases, and CPU-side processing. We present Harmonia, an end-to-end RAG serving framework that addresses these bottlenecks through (i) a flexible pipeline specification interface for composing custom workflows, (ii) heterogeneity-aware deployment that provisions and configures components as a distributed inference system, and (iii) a closed-loop runtime controller that monitors load and execution progress and reduces SLO violations through request prioritization and auto-scaling. Across four RAG applications, Harmonia outperforms commercial alternatives, improving throughput by more than 2.04x while reducing SLO violations by up to 78.4 percent.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 35 of its 73 references that are in the map. First posted on arXiv 1 May 2025.

Builds on

35 of 73 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving18 Mar
  2. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval28 Feb
  3. 2025Fast or Better? Balancing Accuracy and Cost in Retrieval-Augmented Generation with Flexible User Control17 Feb
  4. 2025CFT-RAG: An Entity Tree Based Retrieval Augmented Generation Algorithm With Cuckoo Filter25 Jan
  5. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting11 Jul
  6. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research22 May
  7. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models15 Oct

Other references · 28

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  8. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  10. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  11. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 172
  12. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
16 more
  1. 2020Pre-training via Paraphrasingcited by 92
  2. 2024Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generationcited by 90
  3. 2024Scaling Retrieval-Based Language Models with a Trillion-Token Datastorecited by 56
  4. 2015MLlib: Machine Learning in Apache Sparkcited by 55
  5. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  6. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  7. 2025SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Modelcited by 32
  8. 2025HopRAG: Multi-Hop Reasoning for Logic-Aware Retrieval-Augmented Generationcited by 24
  9. 2024T-RAG: Lessons from the LLM Trenchescited by 23
  10. 2024ChunkRAG: Novel LLM-Chunk Filtering Method for RAG Systemscited by 23
  11. 2020LazyBatching: An SLA-aware Batching System for Cloud Machine Learning Inferencecited by 22
  12. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  13. 2024RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generationcited by 16
  14. 2024HiQA: A Hierarchical Contextual Augmentation RAG for Multi-Documents QAcited by 6
  15. 2023MOSEL: Inference Serving Using Dynamic Modality Selectioncited by 5
  16. 2025SOSecure: Safer Code Generation with RAG and StackOverflow Discussionscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
  2. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  3. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation
  4. 2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systems
  5. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  6. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  7. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement
  8. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation