mlsanityData to 14 Oct 2026

Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs

arXiv 2609.17564 ↗First posted 23 Jul 2026cs.LG cs.IRcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

also in 1 more
Abstract (arXiv)

Deploying retrieval-augmented generation (RAG) on commodity GPUs such as the NVIDIA T4 (16 GB VRAM) exposes a practical failure mode we call the Compression Paradox: neural prompt compression can add key-value (KV) cache contention and preprocessing latency that outweigh generation-time savings, while skipping compression can cause out-of-memory (OOM) failures on long contexts. We identify two distinct failure mechanisms when a vLLM-served LLM and a PyTorch-based compressor are co-deployed under tight memory budgets, and introduce the Tri-Metric Router, a deterministic, training-free policy that selects among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines. The router uses three CPU-side signals: spatial complexity (L), syntactic density (ρkey), and type-token ratio (TTR). Unlike prior semantic-only adaptation, our dispatch signal is hardware-physical, based on VRAM headroom and a latency crossover point. Thresholds are calibrated from profiling on LongBench qasper, yielding an operating crossover near 4,332 words on T4; our contribution is this calibration methodology rather than a hardware-specific constant. On out-of-distribution holdouts, the method achieves 0% OOM failures, 88.5 ± 4.4% oracle alignment, and 49.3% Combined F1, improving over always-on lexical compression by 5.2 points without additional VRAM or training cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 18 of its 25 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

18 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach23 Jul

Other references · 17

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  8. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  10. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
5 more
  1. 2023Active Retrieval Augmented Generationcited by 516
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  3. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  4. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  5. 2024In Defense of RAG in the Era of Long-Context Language Modelscited by 53

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
  2. 2026RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference
  3. 2026FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
  4. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
  5. 2025VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
  6. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  7. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
  8. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains