mlsanityData to 14 Oct 2026

BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inference

arXiv 2609.13592 ↗First posted 11 Sep 2026cs.DC cs.AR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

GPU memory bandwidth and capacity limit throughput in large language model (LLM) inference. The GPU memory system consists of a primary tier of high-bandwidth memory (HBM) and a secondary tier of host memory connected via CPU-to-GPU interconnect. Current serving systems treat the tiers hierarchically: they serve exclusively from HBM when data fits, and otherwise prefetch data from host memory to HBM before use. In both cases, the host memory bandwidth is never well utilized. Prefetching expands capacity by utilizing host memory, but consumes HBM bandwidth for writes, reducing the bandwidth available for demand loads. We observe that fully utilizing both host and HBM bandwidth requires each wave of GPU threadblocks to access both tiers concurrently and in proportion to their bandwidth ratio. Existing bandwidth-proportional placement strategies fail to provide concurrency because they are not aware of GPU waves, and the large 2MB GPU page size. This paper presents BOOST, the first runtime system that provides concurrent and proportional access to both GPU memory tiers, extracting the combined bandwidth of host memory and HBM for LLM inference without kernel changes. The key insight in BOOST is to use kernel access patterns to make page allocation and runtime data management wave-aware. For static model weights, BOOST applies modulo-based page placement that eliminates access-ratio variance; for dynamically provisioned attention key-value (KV) pairs, it makes the free KV page pool wave-aware. We integrate BOOST into vLLM and evaluate on a Grace Hopper system. At iso-batch size, BOOST improves Time-per-Output-Token (TPOT) by 4.3% over HBM-only serving, whereas prefetching degrades TPOT by 6%. In high-throughput serving, BOOST improves throughput by 31% on average, outperforming prefetching by 15%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 21 of its 62 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

21 of 62 references.

References · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  10. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  12. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
9 more
  1. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  3. 2024Mirage: A Multi-Level Superoptimizer for Tensor Programscited by 41
  4. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  5. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 25
  6. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  7. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 14
  8. 2024Pie: Pooling CPU Memory for LLM Inferencecited by 13
  9. 2024Understanding Data Movement in Tightly Coupled Heterogeneous Systems: A Case Study with the Grace Hopper Superchipcited by 8

Built on by

Not cited yet.