# MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG

arXiv: https://arxiv.org/abs/2507.01079 · HTML: https://mlsanity.com/paper/2507.01079/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 27 of its 57 references that are in the map.
- First posted on arXiv 1 Jul 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 27 of 27)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms](https://mlsanity.com/paper/1807.05614/) (2018)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Search](https://mlsanity.com/paper/2101.12631/) (2021)
- [Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks](https://mlsanity.com/paper/2407.21059/) (2024)
- [Hindsight: Posterior-guided training of retrievers for improved open-ended generation](https://mlsanity.com/paper/2110.07752/) (2021)
- [EFANNA : An Extremely Fast Approximate Nearest Neighbor Search Algorithm Based on kNN Graph](https://mlsanity.com/paper/1609.07228/) (2016)
- [MeMemo: On-device Retrieval Augmentation for Private and Personalized Text Generation](https://mlsanity.com/paper/2407.01972/) (2024)
- [EdgeRAG: Online-Indexed RAG for Edge Devices](https://mlsanity.com/paper/2412.21023/) (2024)

## Built on by (most cited first, 11 of 11)

- [Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies](https://mlsanity.com/paper/2603.16952/) (2026)
- [Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Execution](https://mlsanity.com/paper/2510.15312/) (2025)
- [From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG](https://mlsanity.com/paper/2605.18271/) (2026)
- [Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging](https://mlsanity.com/paper/2608.05541/) (2026)
- [A Unified Model and Document Representation for On-Device Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.14403/) (2026)
- [Accelerating AI-Powered Research: The PuppyChatter Framework for Usable and Flexible Tooling](https://mlsanity.com/paper/2605.17809/) (2026)
- [One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA](https://mlsanity.com/paper/2606.10572/) (2026)
- [From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG](https://mlsanity.com/paper/2608.19535/) (2026)
- [TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching](https://mlsanity.com/paper/2609.21172/) (2026)
- [Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs](https://mlsanity.com/paper/2609.34727/) (2026)
- [PhaseGate: Phase-Aware CPU Retrieval Scheduling for On-Device LLMs on Unified Memory](https://mlsanity.com/paper/2610.04537/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has proven effective on server infrastructures, but its application on mobile devices is still underexplored due to limited memory and power resources. Existing vector search and RAG solutions largely assume abundant computation resources, making them impractical for on-device scenarios. In this paper, we propose MobileRAG, a fully on-device pipeline that overcomes these limitations by combining a mobile-friendly vector search algorithm, textitEcoVector, with a lightweight textitSelective Content Reduction (SCR) method. By partitioning and partially loading index data, EcoVector drastically reduces both memory footprint and CPU usage, while the SCR method filters out irrelevant text to diminish Language Model (LM) input size without degrading accuracy. Extensive experiments demonstrated that MobileRAG significantly outperforms conventional vector search and RAG methods in terms of latency, memory usage, and power consumption, while maintaining accuracy and enabling offline operation to safeguard privacy in resource-constrained environments.