mlsanityData to 7 Oct 2026

xLLM Technical Report

arXiv 2510.14686 ↗First posted 16 Oct 2025cs.DC cs.AIcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework designed for high-performance, large-scale enterprise-grade serving, with deep optimizations for diverse AI accelerators. To address these challenges, xLLM builds a novel decoupled service-engine architecture. At the service layer, xLLM-Service features an intelligent scheduling module that efficiently processes multimodal requests and co-locates online and offline tasks through unified elastic scheduling to maximize cluster utilization. This module also relies on a workload-adaptive dynamic Prefill-Decode (PD) disaggregation policy and a novel Encode-Prefill-Decode (EPD) disaggregation policy designed for multimodal inputs. Furthermore, it incorporates a distributed architecture to provide global KV Cache management and robust fault-tolerant capabilities for high availability. At the engine layer, xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources. This is achieved through comprehensive multi-layer execution pipeline optimizations, an adaptive graph mode and an xTensor memory management. xLLM-Engine also further integrates algorithmic enhancements such as optimized speculative decoding and dynamic EPLB, collectively serving to substantially boost throughput and inference efficiency. Extensive evaluations demonstrate that xLLM delivers significantly superior performance and resource efficiency. Under identical TPOT constraints, xLLM achieves throughput up to 1.7x that of MindIE and 2.2x that of vLLM-Ascend with Qwen-series models, while maintaining an average throughput of 1.7x that of MindIE with Deepseek-series models. xLLM framework is publicly available at https://github.com/jd-opensource/xllm and https://github.com/jd-opensource/xllm-service.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 58 of its 90 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

58 of 90 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning6 May
  2. 2025Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving1 Mar
  3. 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location15 Jan
  4. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughput22 Aug
  5. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  6. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  7. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  12. 2023A Survey on Multimodal Large Language Modelscited by 898
38 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  7. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 323
  8. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  9. 2023Recommender Systems with Generative Retrievalcited by 191
  10. 2024Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendationscited by 184
  11. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  12. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  13. 2023Generative Recommendation: Towards Next-generation Recommender Paradigmcited by 69
  14. 2024HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modelingcited by 63
  15. 2024EAGER: Two-Stream Generative Recommender with Behavior-Semantic Collaborationcited by 55
  16. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 52
  17. 2025MTGR: Industrial-Scale Generative Recommendation Framework in Meituancited by 47
  18. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  19. 2025Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representationscited by 35
  20. 2024Efficient Training of Large Language Models on Distributed Infrastructures: A Surveycited by 26
  21. 2025OneRec Technical Reportcited by 24
  22. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 22
  23. 2024Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPUcited by 20
  24. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  25. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  26. 2025Robust LLM Training Infrastructure at ByteDancecited by 14
  27. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 13
  28. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  29. 2023TRANSOM: An Efficient Fault-Tolerant System for Training LLMscited by 12
  30. 2025TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inferencecited by 11
  31. 2025Efficient Beam Search for Large Language Models Using Trie-Based Decodingcited by 10
  32. 2025PinRec: Unified Generative Retrieval for Pinterest Recommender Systemscited by 8
  33. 2023LLM-based Interaction for Content Generation: A Case Study on the Perception of Employees in an IT departmentcited by 6
  34. 2025EcoServe: Enabling Cost-effective LLM Serving with Proactive Intra- and Inter-Instance Orchestrationcited by 4
  35. 2025eLLM: Elastic Memory Management Framework for Efficient LLM Servingcited by 4
  36. 2024ISO: Overlap of Computation and Communication within Seqenence For LLM Inferencecited by 3
  37. 2025GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Modelscited by 3
  38. 2025FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2026RTPrune: Reading-Twice Inspired Token Pruning for Efficient DeepSeek-OCR Inferencecited by 4
  2. 2025xGR: Efficient Generative Recommendation Serving at Scalecited by 2
  3. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality
  2. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  3. 2025Hybrid Offline-online Scheduling Method for Large Language Model Inference Optimization
  4. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
  5. 2025SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference
  6. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure
  7. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
  8. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture