mlsanityData to 7 Oct 2026

HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures

arXiv 2602.00748 ↗First posted 31 Jan 2026cs.DC cs.AI cs.ARcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

The rapid evolution of Large Language Models (LLMs) towards long-context reasoning and sparse architectures has pushed memory requirements far beyond the capacity of individual device HBM. While emerging supernode architectures offer terabyte-scale shared memory pools via high-bandwidth interconnects, existing software stacks fail to exploit this hardware effectively. Current runtime-based offloading and swapping techniques operate with a local view, leading to reactive scheduling and exposed communication latency that stall the computation pipeline. In this paper, we propose the SuperNode Memory Management Framework (HyperOffload). It employs a compiler-assisted approach that leverages graph-driven memory management to treat remote memory access as explicit operations in the computation graph, specifically designed for hierarchical SuperNode architectures. Unlike reactive runtime systems, SuperNode represents data movement using cache operators within the compiler's Intermediate Representation (IR). This design enables a global, compile-time analysis of tensor lifetimes and execution dependencies. Leveraging this visibility, we develop a global execution-order refinement algorithm that statically schedules data transfers to hide remote memory latency behind compute-intensive regions. We implement SuperNode within the production deep learning framework MindSpore, adding a remote memory backend and specialized compiler passes. Evaluation on representative LLM workloads shows that SuperNode reduces peak device memory usage by up to 26% for inference while maintaining end-to-end performance. Our work demonstrates that integrating memory-augmented hardware into the compiler's optimization framework is essential for scaling next-generation AI workloads.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 39 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

27 of 39 references.

References · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
15 more
  1. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  4. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  5. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  6. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  7. 2020Sparse Weight Activation Trainingcited by 45
  8. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 22
  9. 2020Memory Optimization for Deep Networkscited by 15
  10. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
  11. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Servingcited by 9
  12. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantizationcited by 5
  13. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantizationcited by 3
  14. 2025STAlloc: Enhancing Memory Efficiency in Large-Scale Model Training with Spatio-Temporal Planningcited by 2
  15. 2025FengHuang: Next-Generation Memory Orchestration for AI Inferencingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  2. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips
  3. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  4. 2025Scaling Intelligence: Designing Data Centers for Next-Gen Language Models
  5. 2026HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution
  6. 2026MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
  7. 2025MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models
  8. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing