mlsanityData to 7 Oct 2026

Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference

arXiv 2603.29002 ↗First posted 30 Mar 2026cs.DC cs.AIcited by 0

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Modern large language models (LLMs) increasingly depends on efficient long-context processing and generation mechanisms, including sparse attention, retrieval-augmented generation (RAG), and compressed contextual memory, to support complex reasoning. We show that these optimizations can be unified into a four-step memory processing pipeline: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference. Through systematic profiling, we identify a 22%-97% memory processing overhead in LLM inference and strong heterogeneity in its computational characteristics. Motivated by this insight, we argue that heterogeneous systems are well-suited to accelerate memory processing and thus end-to-end inference. We demonstrate this approach on a GPU-FPGA system by offloading sparse, irregular, and memory-bounded operations to FPGAs while retaining compute-intensive operations on GPUs. Evaluated on an AMD MI210 GPU and an Alveo U55C FPGA, our system is up to 2.2× faster and achieves up to 4.7× less energy across multiple LLM inference optimizations than the GPU baseline (similar results hold on NVIDIA A100). These results establish heterogeneous systems as a practical direction for efficient LLM memory processing and inform future heterogeneous hardware design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 48 of its 61 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

48 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking31 Oct
  2. 2023Active Retrieval Augmented Generation11 May
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 45

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2020Reformer: The Efficient Transformercited by 1,658
33 more
  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  5. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  8. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  9. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  10. 2024Titans: Learning to Memorize at Test Timecited by 303
  11. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  12. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  13. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  14. 2022Recurrent Memory Transformercited by 197
  15. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  16. 2025Test-Time Training Done Rightcited by 121
  17. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 119
  18. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 87
  19. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  20. 2025TileLang: A Composable Tiled Programming Model for AI Systemscited by 42
  21. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  22. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  23. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  24. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  25. 2025QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Managementcited by 24
  26. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  27. 2022HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Rerankingcited by 19
  28. 2024Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAGcited by 16
  29. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  30. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAscited by 4
  31. 2025InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNscited by 3
  32. 2026FlexLLM: Composable HLS Library for Flexible Hybrid LLM Accelerator Designcited by 3
  33. 2021Serpens: A High Bandwidth Memory Based Accelerator for General-Purpose Sparse Matrix-Vector Multiplicationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  2. 2024Inference Performance Optimization for Large Language Models on CPUs
  3. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  4. 2025InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
  5. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  6. 2023Efficient LLM inference solution on Intel GPU
  7. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  8. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling