# From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG

arXiv: https://arxiv.org/abs/2608.19535 · HTML: https://mlsanity.com/paper/2608.19535/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)
- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research](https://mlsanity.com/paper/2405.13576/) (2024)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs](https://mlsanity.com/paper/2406.15319/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [A Review on Edge Large Language Models: Design, Execution, and Applications](https://mlsanity.com/paper/2410.11845/) (2024)
- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [Provence: efficient and robust context pruning for retrieval-augmented generation](https://mlsanity.com/paper/2501.16214/) (2025)
- [VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting](https://mlsanity.com/paper/2507.05116/) (2025)
- [METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation](https://mlsanity.com/paper/2412.10543/) (2024)
- [EdgeRAG: Online-Indexed RAG for Edge Devices](https://mlsanity.com/paper/2412.21023/) (2024)
- [Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architectures](https://mlsanity.com/paper/2405.04700/) (2024)
- [MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG](https://mlsanity.com/paper/2507.01079/) (2025)
- [Cross-Platform Scaling of Vision-Language-Action Models from Edge to Cloud GPUs](https://mlsanity.com/paper/2509.11480/) (2025)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.