mlsanityData to 14 Oct 2026

From Retrieved Context to Runtime Control: Adaptive Compression for Edge-based RAG

arXiv 2608.19535 ↗First posted 20 Aug 2026cs.AI cs.CL cs.DCcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

also in 1 more
Abstract (arXiv)

Retrieval-augmented generation (RAG) improves language-model responses by grounding generation in external passages, which comes with overhead: retrieved context lengthens the prompt, increasing prefill work, KV-cache footprint, memory traffic, latency, and energy. Context compression offers a natural remedy by pruning retrieved text before generation. However, state-of-the-art context-compression methods are typically used with a fixed compression budget, or with the rate selected offline and then applied at inference time. This static view ignores both workload variation and the live state of the edge device. On an edge SoC, compression is not free: the compressor itself runs on the same SoC and consumes latency and energy that can offset any generation savings. This paper proposes a vision for telemetry-informed adaptive compression in edge RAG, grounded in experimental evidence. We characterize the compression tradeoff on the NVIDIA Jetson AGX Thor using Llama and Qwen generators, Natural Questions and HotpotQA datasets, and LLMLingua-2 compression. Our measurements show that generation dominates the RAG budget for larger models, reaching roughly 90% of per-query latency and 91% of GPU energy for 7B-8B generators. Exploring the impact of the compression rate reveals an adaptive operating region: mild compression can miss energy opportunities, and overly aggressive compression can hurt inference quality. Intermediate compression can reduce GPU energy by up to 53.2%, and SoC energy by up to 48.2%, with negligible quality loss. We argue for runtime policies that dynamically manage compression, guided by workload features and edge telemetry.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 53 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

31 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAG1 Jul
  2. 2024EdgeRAG: Online-Indexed RAG for Edge Devices30 Dec
  3. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptation13 Dec
  4. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs21 Jun
  5. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research22 May
  6. 2024Robust Implementation of Retrieval-Augmented Generation on Edge-based Computing-in-Memory Architectures7 May

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2017Billion-scale similarity search with GPUscited by 2,497
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  11. 2019Passage Re-ranking with BERTcited by 645
  12. 2023Learning to Compress Prompts with Gist Tokenscited by 390
13 more
  1. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  2. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  3. 2023Adapting Language Models to Compress Contextscited by 356
  4. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  5. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 256
  6. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  7. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  8. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  9. 2024Prompt Compression for Large Language Models: A Surveycited by 67
  10. 2025Provence: efficient and robust context pruning for retrieval-augmented generationcited by 25
  11. 2025VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Votingcited by 22
  12. 2025Cross-Platform Scaling of Vision-Language-Action Models from Edge to Cloud GPUscited by 8
  13. 2025A Memory-Efficient Retrieval Architecture for RAG-Enabled Wearable Medical LLMs-Agentscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Less Is More: Elevating RAG via Performance-Driven Context Compression
  2. 2025Enhancing RAG Efficiency with Adaptive Context Compression
  3. 2026Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs
  4. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU
  5. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  6. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systems
  7. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  8. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey