# OpRAG: A Resource-Deterministic Runtime for GPU-Backed Multi-Stage RAG Workflows

arXiv: https://arxiv.org/abs/2608.08340 · HTML: https://mlsanity.com/paper/2608.08340/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 8 Aug 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 26 of 26)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation](https://mlsanity.com/paper/2409.05591/) (2024)
- [Parsl: Pervasive Parallel Programming in Python](https://mlsanity.com/paper/1905.02158/) (2019)
- [Exploration of LLM Multi-Agent Application Implementation Based on LangGraph+CrewAI](https://mlsanity.com/paper/2411.18241/) (2024)
- [Towards Scalable Dataframe Systems](https://mlsanity.com/paper/2001.00888/) (2020)
- [High Performance Data Engineering Everywhere](https://mlsanity.com/paper/2007.09589/) (2020)
- [Design and Performance Characterization of RADICAL-Pilot on Leadership-class Platforms](https://mlsanity.com/paper/2103.00091/) (2021)
- [Supercharging Distributed Computing Environments For High Performance Data Engineering](https://mlsanity.com/paper/2301.07896/) (2023)
- [RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms](https://mlsanity.com/paper/2105.13185/) (2021)
- [High Performance Dataframes from Parallel Processing Patterns](https://mlsanity.com/paper/2209.06146/) (2022)
- [Hybrid Cloud and HPC Approach to High-Performance Dataframes](https://mlsanity.com/paper/2212.13732/) (2022)
- [In-depth Analysis On Parallel Processing Patterns for High-Performance Dataframes](https://mlsanity.com/paper/2307.01394/) (2023)
- [Deep RC: A Scalable Data Engineering and Deep Learning Pipeline](https://mlsanity.com/paper/2502.20724/) (2025)
- [On The Reproducibility Limitations of RAG Systems](https://mlsanity.com/paper/2509.18869/) (2025)
- [Performance comparison of Dask and Apache Spark on HPC systems for Neuroimaging](https://mlsanity.com/paper/2406.01409/) (2024)
- [Tuning LLMs by RAG Principles: Towards LLM-native Memory](https://mlsanity.com/paper/2503.16071/) (2025)
- [RHAPSODY: Execution of Hybrid AI-HPC Workflows at Scale](https://mlsanity.com/paper/2512.20795/) (2025)
- [Higress-RAG: A Holistic Optimization Framework for Enterprise Retrieval-Augmented Generation via Dual Hybrid Retrieval, Adaptive Routing, and CRAG](https://mlsanity.com/paper/2602.23374/) (2025)
- [AAFLOW: Scalable Patterns for Agentic AI Workflows](https://mlsanity.com/paper/2605.02162/) (2026)
- [[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows](https://mlsanity.com/paper/2607.10987/) (2026)

## Built on by (most cited first, 1 of 1)

- [Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control](https://mlsanity.com/paper/2608.12123/) (2026)

## Abstract (arXiv)

Agentic retrieval-augmented generation (RAG) systems combine preprocessing, embedding, retrieval, memory access, context construction, generation, and vector-index updates. Although LLM decoding is GPU-bound, the surrounding orchestration layer can still limit end-to-end performance through serialization overhead, fragmented scheduling, inefficient batching, and CPU--GPU pipeline stalls. Existing frameworks provide flexible control flow, while distributed runtimes provide scalable task parallelism, but neither exposes RAG stages as resource-aware operators with deterministic execution semantics. We present OpRAG, a resource-deterministic distributed runtime for GPU-backed multi-stage RAG workflows. OpRAG models embedding, retrieval, reasoning, memory, and upsert as first-class operators and lowers them into communication-aware execution graphs. It combines an Arrow zero-copy data plane, persistent workers, bounded queues, CPU tokenizer prefetching, batched GPU embedding, and overlapped retrieval/generation execution to reduce non-model overhead around LLM inference. We evaluate OpRAG using Llama3-8B and Mistral-7B with FlashAttention~2, BF16 execution, and 32K RAG chunks. In end-to-end GPU pipeline experiments, OpRAG improves over the nearest competitor by 16.16% for Llama3-8B and 15.66% for Mistral-7B, and over RayScalableRAG by 20.57% and 20.71%, respectively. Against LangChain, LangGraph, CrewAI, and AutoGen, OpRAG is 17.77% and 17.48% faster than the best framework baseline. In Higress-style query serving, OpRAG reduces hybrid retrieval latency by 59.20--59.62% and generation-scenario latency by 52.48--53.55%, while preserving 100% Recall@5. These results show that optimizing the distributed orchestration layer can substantially improve GPU-backed multi-stage RAG without modifying the LLM decoding kernel.