mlsanityData to 7 Oct 2026

HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement

arXiv 2510.20878 ↗First posted 23 Oct 2025cs.LG cs.AIcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) improves model output accuracy by leveraging external knowledge bases, serving as an effective solution to address hallucination issues and knowledge-update delays in Large Language Models (LLMs). However, the introduction of external knowledge bases presents RAG with challenges in long-context processing, significantly increasing memory consumption and inference latency. Existing research accelerates inference by precomputing Key and Value (KV) of the knowledge base and loading them on-demand during inference. Based on the access frequency of different KV chunks within the external knowledge base, this paper proposes a hotness-aware RAG (HA-RAG) inference optimization system. First, leveraging the numerical distribution of KV chunks, we introduce a hotness-aware mixed-precision compressing and loading method to reduce disk I/O and memory access overhead. Second, we design a hotness-aware data placement strategy that prioritizes storing frequently accessed KV chunks in high-speed memory to improve data access efficiency. Experimental results demonstrate that, compared with TurboRAG, the proposed HA-RAG achieves an average speedup of 2.10x and maximum speedup of 10.49x in Time-To-First-Token (TTFT) with negligible accuracy loss.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 58 of its 74 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

58 of 74 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps19 May
  2. 2025RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU17 Apr
  3. 2025Efficient Distributed Retrieval-Augmented Generation for Enhancing Language Model Performance15 Apr
  4. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval28 Feb
  5. 2025HawkBench: Investigating Resilience of RAG Methods on Stratified Information-Seeking Tasks19 Feb
  6. 2025Does RAG Really Perform Bad For Long-Context Processing?17 Feb
  7. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models10 May

Other references · 51

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
38 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  2. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  3. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  4. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 310
  5. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  6. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  7. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  8. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  9. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  10. 2022FP8 Formats for Deep Learningcited by 187
  11. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  12. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  13. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  14. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  15. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  16. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  17. 2023Natural Language Reasoning, A Surveycited by 61
  18. 2024TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Textcited by 60
  19. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  20. 2023Do Large Language Models Know about Facts?cited by 49
  21. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  22. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 45
  23. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  24. 2025Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generationcited by 36
  25. 2024Block-Attention for Efficient Prefillingcited by 34
  26. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  27. 2025APE: Faster and Longer Context-Augmented Generation via Adaptive Parallel Encodingcited by 29
  28. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  29. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  30. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reusecited by 22
  31. 2024SwiftKV: Fast Prefill-Optimized Inference with Knowledge-Preserving Model Transformationcited by 18
  32. 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
  33. 2024Improving Factuality with Explicit Working Memorycited by 10
  34. 2025HyperRAG: Enhancing Quality-Efficiency Tradeoffs in Retrieval-Augmented Generation with Reranker KV-Cache Reusecited by 8
  35. 2025Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Modelscited by 5
  36. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Headscited by 4
  37. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniquescited by 4
  38. 2025Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMscited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving
  2. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  3. 2024Accelerating Retrieval-Augmented Generation
  4. 2025RAGPulse: An Open-Source RAG Workload Trace to Optimize RAG Serving Systems
  5. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
  6. 2025Harmonia: End-to-End RAG Serving Optimization
  7. 2024Inference Scaling for Long-Context Retrieval Augmented Generation
  8. 2026RAG-Stack: Co-Optimizing RAG Serving Performance and Quality