mlsanityData to 7 Oct 2026

NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement

arXiv 2607.26633 ↗First posted 29 Jul 2026cs.ARcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern LLMs and their agentic applications are broadening the range of serving workloads, spanning context lengths from a few hundred tokens to hundreds of thousands. As these requests frequently interleave within the same serving window, LLM serving systems must handle highly heterogeneous mixed-length workloads. Such mixed-length workloads expose fundamental inefficiencies in GPU-centric serving architectures, whose throughput depends on large, memory-constrained batches. In this paper, we present NELSSA, an LLM serving system that integrates GPUs with real-world Processing-near-Memory (PNM) accelerator devices to efficiently support mixed-length workloads. NELSSA employs length-based request placement to route short-context requests to GPUs and long-context requests to the PNM tier, incorporating runtime migration to accommodate dynamic context growth without recomputation. We prototype NELSSA as an end-to-end system, implementing device-level sparse attention on PNM, GPU decode kernels, and a host-side runtime that orchestrates scheduling and cross-tier memory movement over a CXL-enabled infrastructure with RPC and RDMA support. Across mixed-length LLM workloads, NELSSA improves decode throughput by up to 5.5x in tokens/sec and reduces P99 latency by up to 15x compared to GPU-only baselines. Our end-to-end prototype and experimental results suggest that integrated GPU-PNM serving, enabled by CXL-based disaggregation, is a promising system paradigm for scalable and flexible LLM infrastructures that support evolving workloads.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 62 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

38 of 62 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  2. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production15 May
  3. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor14 May
  4. 2025HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing18 Apr
  5. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  6. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  11. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 27

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  10. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  11. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  12. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
15 more
  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  2. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  3. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
  4. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  5. 2025Budget-Aware Tool Use Enables Effective Agent Scalingcited by 42
  6. 2024Don't Stop Me Now: Embedding Based Scheduling for LLMscited by 34
  7. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 32
  8. 2026Fast KV Compaction via Attention Matchingcited by 28
  9. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  10. 2025Evaluating Long-Context Reasoning in LLM-Based WebAgentscited by 15
  11. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 12
  12. 2026Challenges and Research Directions for Large Language Model Inference Hardwarecited by 10
  13. 2025Make LLM Inference Affordable to Everyone: Augmenting GPU Memory with NDP-DIMMcited by 6
  14. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  15. 2026Can LLMs Track Their Output Length? A Dynamic Feedback Mechanism for Precise Length Regulationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  2. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing
  3. 2026RAPID-Serve: Resource-efficient and Accelerated P/D Intra-GPU Disaggregation
  4. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  5. 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
  6. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  7. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  8. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure