mlsanityData to 7 Oct 2026

ODMA: On-Demand Memory Allocation Strategy for LLM Serving on LPDDR-Class Accelerators

arXiv 2512.09427 ↗First posted 10 Dec 2025cs.AR cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Existing memory management techniques severely hinder efficient Large Language Model serving on accelerators constrained by poor random-access bandwidth.While static pre-allocation preserves memory contiguity,it incurs significant overhead due to worst-case provisioning.Conversely,fine-grained paging mitigates this overhead but relies on HBM's high random-access tolerance, making it unsuitable for LPDDR systems where non-sequential access rapidly degrades bandwidth. Furthermore, prior works typically assume static distributions and HBM characteristics, thereby failing to resolve the critical fragmentation and bandwidth constraints inherent to LPDDR hardware. We present ODMA, an on-demand memory allocation strategy tailored for random-access-constrained accelerators, such as the Cambricon MLU series.ODMA advances generation-length prediction by addressing two critical limitations in production workloads: (i) distribution drift that invalidates static bucket boundaries, and (ii) performance fragility under heavy-tailed request patterns. ODMA integrates a lightweight length predictor with adaptive bucket partitioning and a fallback safety pool. Bucket boundaries are dynamically recalibrated via online histograms to maximize utilization, while the safety pool ensures robustness against prediction errors. On Alpaca and Google-NQ benchmarks, ODMA improves S3's prediction accuracy from 98.60% to 99.55% and 82.68% to 93.36%, respectively. Deployment with DeepSeek-R1-Distill-Qwen-7B on Cambricon MLU370-X4 accelerators demonstrates that ODMA increases KV-cache utilization by up to 19.25% (absolute) and throughput (TPS) by 23-27% over static baselines, validating the efficacy of predictor-driven contiguous allocation for LPDDR-class devices.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 9 of its 17 references that are in the map. First posted on arXiv 10 Dec 2025.

Builds on

9 of 17 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  2. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  3. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 6

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  6. 2022Unveiling the Real Performance of LPDDR5 Memoriescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  2. 2026SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving
  3. 2026HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Accelerators
  4. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
  5. 2026Hardware-Software Co-design for 3D-DRAM-based LLM Serving Accelerator
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management
  8. 2026HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution