# AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains

arXiv: https://arxiv.org/abs/2407.21255 · HTML: https://mlsanity.com/paper/2407.21255/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2024.

## Built on by (most cited first, 14 of 14)

- [Fast Inference for Augmented Large Language Models](https://mlsanity.com/paper/2410.18248/) (2024)
- [A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design](https://mlsanity.com/paper/2607.09084/) (2026)
- [SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips](https://mlsanity.com/paper/2601.20309/) (2026)
- [Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference](https://mlsanity.com/paper/2602.00328/) (2026)
- [Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model](https://mlsanity.com/paper/2602.07878/) (2026)
- [SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling](https://mlsanity.com/paper/2607.08565/) (2026)
- [PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization](https://mlsanity.com/paper/2607.16184/) (2026)
- [SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer](https://mlsanity.com/paper/2607.28150/) (2026)
- [PASCAL: A Phase-Aware Scheduling Algorithm for Serving Reasoning-based Large Language Models](https://mlsanity.com/paper/2602.11530/) (2026)
- [Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC](https://mlsanity.com/paper/2604.07609/) (2026)
- [APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs](https://mlsanity.com/paper/2506.03296/) (2025)
- [Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs](https://mlsanity.com/paper/2607.23115/) (2026)
- [DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs](https://mlsanity.com/paper/2607.28848/) (2026)
- [EMA: Elastic and Performance Transparent Memory Across GPUs](https://mlsanity.com/paper/2609.27040/) (2026)

## Abstract (arXiv)

Inference on large-language models (LLMs) is constrained by GPU memory capacity. A sudden increase in the number of inference requests to a cloud-hosted LLM can deplete GPU memory, leading to contention between multiple prompts for limited resources. Modern LLM serving engines deal with the challenge of limited GPU memory using admission control, which causes them to be unresponsive during request bursts. We propose that preemptive scheduling of prompts in time slices is essential for ensuring responsive LLM inference, especially under conditions of high load and limited GPU memory. However, preempting prompt inference incurs a high paging overhead, which reduces inference throughput. We present Aqua, a GPU memory management framework that significantly reduces the overhead of paging inference state, achieving both responsive and high-throughput inference even under bursty request patterns. We evaluate Aqua by hosting several state-of-the-art large generative ML models of different modalities on servers with 8 Nvidia H100 80G GPUs. Aqua improves the responsiveness of LLM inference by 20X compared to the state-of-the-art and improves LLM inference throughput over a single long prompt by 4X.