mlsanityData to 7 Oct 2026

MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference

arXiv 2603.26557 ↗First posted 27 Mar 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

35 of 44 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
23 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  7. 2024The Faiss librarycited by 370
  8. 2026Ministral 3cited by 88
  9. 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 67
  10. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
  11. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 56
  12. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
  13. 2024How Much Can RAG Help the Reasoning of LLM?cited by 31
  14. 2025MiniRAG: Towards Extremely Simple Retrieval-Augmented Generationcited by 28
  15. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Modelscited by 22
  16. 2024GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Cachingcited by 21
  17. 2024SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Modelscited by 16
  18. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  19. 2024MeanCache: User-Centric Semantic Caching for LLM Web Servicescited by 14
  20. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  21. 2025Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptationcited by 12
  22. 2025Category-Aware Semantic Caching for Heterogeneous LLM Workloadscited by 4
  23. 2025Targeting Misalignment: A Conflict-Aware Framework for Reward-Model-based LLM Alignmentcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating LLM Inference with Precomputed Query Storage
  2. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
  3. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
  4. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  5. 2025Tuning LLMs by RAG Principles: Towards LLM-native Memory
  6. 2025MemR3: Memory Retrieval via Reflective Reasoning for LLM Agents
  7. 2023SparQ Attention: Bandwidth-Efficient LLM Inference
  8. 2025RAG-R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism