mlsanityData to 7 Oct 2026

Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models

arXiv 2505.11271 ↗First posted 16 May 2025cs.CL cs.AI cs.IRcited by 5

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 29 of its 44 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

29 of 44 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
17 more
  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  2. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  3. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  4. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  5. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  7. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  8. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  9. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  10. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  11. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  12. 2019Correlation Coefficients and Semantic Textual Similaritycited by 20
  13. 2024SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Modelscited by 16
  14. 2025IC-Cache: Efficient Large Language Model Serving via In-context Cachingcited by 13
  15. 2024InstCache: A Predictive Cache for LLM Servingcited by 6
  16. 2023Hybrid-RACA: Hybrid Retrieval-Augmented Composition Assistance for Real-time Text Predictioncited by 4
  17. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Cachingcited by 3
  2. 2026TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agentscited by 2
  3. 2025PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devicescited by 1
  4. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  5. 2026Mobility-Aware Cache Framework for Scalable LLM-Based Human Mobility Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CacheRAG: A Semantic Caching System for Retrieval-Augmented Generation in Knowledge Graph Question Answering
  2. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
  3. 2025Tagging-Augmented Generation: Assisting Language Models in Finding Intricate Knowledge In Long Contexts
  4. 2025LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
  5. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
  6. 2025Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs
  7. 2025Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k
  8. 2025T2: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering