# EdgeRAG: Online-Indexed RAG for Edge Devices

arXiv: https://arxiv.org/abs/2412.21023 · HTML: https://mlsanity.com/paper/2412.21023/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 5 of its 30 references that are in the map.
- First posted on arXiv 30 Dec 2024.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 5 of 5)

- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text](https://mlsanity.com/paper/2210.02928/) (2022)
- [Retrieval Augmented Visual Question Answering with Outside Knowledge](https://mlsanity.com/paper/2210.03809/) (2022)

## Built on by (most cited first, 13 of 13)

- [MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG](https://mlsanity.com/paper/2507.01079/) (2025)
- [Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Execution](https://mlsanity.com/paper/2510.15312/) (2025)
- [LEANN: A Low-Storage Vector Index](https://mlsanity.com/paper/2506.08276/) (2025)
- [Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents](https://mlsanity.com/paper/2609.22486/) (2026)
- [LEKA:LLM-Enhanced Knowledge Augmentation](https://mlsanity.com/paper/2501.17802/) (2025)
- [CaGR-RAG: Context-aware Query Grouping for Disk-based Vector Search in RAG Systems](https://mlsanity.com/paper/2505.01164/) (2025)
- [WebANNS: Fast and Efficient Approximate Nearest Neighbor Search in Web Browsers](https://mlsanity.com/paper/2507.00521/) (2025)
- [PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devices](https://mlsanity.com/paper/2601.11553/) (2025)
- [DGRAG: Distributed Graph-based Retrieval-Augmented Generation in Edge-Cloud Systems](https://mlsanity.com/paper/2505.19847/) (2025)
- [CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation](https://mlsanity.com/paper/2510.19670/) (2025)
- [CoEdge-RAG: Optimizing Hierarchical Scheduling for Retrieval-Augmented LLMs in Collaborative Edge Computing](https://mlsanity.com/paper/2511.05915/) (2025)
- [From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG](https://mlsanity.com/paper/2608.19535/) (2026)
- [RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2609.05760/) (2026)

## Abstract (arXiv)

Deploying Retrieval Augmented Generation (RAG) on resource-constrained edge devices is challenging due to limited memory and processing power. In this work, we propose EdgeRAG which addresses the memory constraint by pruning embeddings within clusters and generating embeddings on-demand during retrieval. To avoid the latency of generating embeddings for large tail clusters, EdgeRAG pre-computes and stores embeddings for these clusters, while adaptively caching remaining embeddings to minimize redundant computations and further optimize latency. The result from BEIR suite shows that EdgeRAG offers significant latency reduction over the baseline IVF index, but with similar generation quality while allowing all of our evaluated datasets to fit into the memory.