# PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devices

arXiv: https://arxiv.org/abs/2601.11553 · HTML: https://mlsanity.com/paper/2601.11553/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 31 of its 55 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)
- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 30 of 31)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion](https://mlsanity.com/paper/2405.16444/) (2024)
- [CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving](https://mlsanity.com/paper/2310.07240/) (2023)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [PowerInfer-2: Fast Large Language Model Inference on a Smartphone](https://mlsanity.com/paper/2406.06282/) (2024)
- [Stable LM 2 1.6B Technical Report](https://mlsanity.com/paper/2402.17834/) (2024)
- [ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition](https://mlsanity.com/paper/2402.15220/) (2024)
- [Fast On-device LLM Inference with NPUs](https://mlsanity.com/paper/2407.05858/) (2024)
- [A Survey of Large Language Model Agents for Question Answering](https://mlsanity.com/paper/2503.19213/) (2025)
- [Sleep-time Compute: Beyond Inference Scaling at Test-time](https://mlsanity.com/paper/2504.13171/) (2025)
- [PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design](https://mlsanity.com/paper/2403.05676/) (2024)
- [Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection](https://mlsanity.com/paper/2405.16178/) (2024)
- [Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models](https://mlsanity.com/paper/2310.09949/) (2023)
- [Accelerating Retrieval-Augmented Language Model Serving with Speculation](https://mlsanity.com/paper/2401.14021/) (2024)
- [BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching](https://mlsanity.com/paper/2411.16102/) (2024)
- [RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving](https://mlsanity.com/paper/2503.14649/) (2025)
- [Theoretical Analysis of Byte-Pair Encoding](https://mlsanity.com/paper/2411.08671/) (2024)
- [EdgeRAG: Online-Indexed RAG for Edge Devices](https://mlsanity.com/paper/2412.21023/) (2024)
- [Pre3: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation](https://mlsanity.com/paper/2506.03887/) (2025)
- [EnronQA: Towards Personalized RAG over Private Documents](https://mlsanity.com/paper/2505.00263/) (2025)
- [PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training](https://mlsanity.com/paper/2411.05046/) (2024)
- [Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts](https://mlsanity.com/paper/2405.01121/) (2024)
- [Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models](https://mlsanity.com/paper/2505.11271/) (2025)
- [LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts](https://mlsanity.com/paper/2501.05554/) (2025)

## Built on by (most cited first, 1 of 1)

- [ArenaFlow: From Trajectory Ranking to Hierarchical Credit Propagation for Open-Ended Agent RL](https://mlsanity.com/paper/2609.21378/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) has been extensively used as a de facto paradigm in various large language model (LLM)-driven applications on mobile devices, such as mobile assistants leveraging personal emails or meeting records. However, due to the lengthy prompts and the resource constraints, mobile RAG systems exhibit significantly high response latency. On this issue, one promising approach is to reuse intermediate computational results across different queries to eliminate redundant computation. But most existing approaches, such as KV cache reuse and semantic cache reuse, are designed for cloud settings and perform poorly, overlooking the distinctive characteristics of mobile RAG. We propose PerCache, a novel hierarchical cache solution designed for reducing end-to-end latency of personalized RAG applications on mobile platforms. PerCache adopts a hierarchical architecture that progressively matches similar queries and QKV cache to maximize the reuse of intermediate results at different computing stages. To improve cache hit rate, PerCache applies a predictive method to populate cache with queries that are likely to be raised in the future. In addition, PerCache can adapt its configurations to dynamic system loads, aiming at maximizing the caching utility with minimal resource consumption. We implement PerCache on top of an existing mobile LLM inference engine with commodity mobile phones. Extensive evaluations show that PerCache can surpass the best-performing baseline by 34.4% latency reduction across various applications and maintain optimal latency performance under dynamic resource changes.