mlsanityData to 14 Oct 2026

OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency

arXiv 2511.22481 ↗First posted 27 Nov 2025cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large Language Models drive a wide range of modern AI applications but impose substantial challenges on large-scale serving systems due to intensive computation, strict latency constraints, and throughput bottlenecks. We introduce OmniInfer, a unified system-level acceleration framework designed to maximize end-to-end serving efficiency through fine-grained optimization of expert placement, cache compression, and scheduling. OmniInfer integrates three complementary components: OmniPlacement for load-aware Mixture-of-Experts scheduling, OmniAttn for sparse attention acceleration, and OmniProxy for disaggregation-aware request scheduling. Built atop vLLM, OmniInfer delivers system-wide performance gains through adaptive resource disaggregation, efficient sparsity exploitation, and global coordination across prefill and decode phases. Evaluated on DeepSeek-R1 within a 10-node Ascend 910C cluster, OmniInfer achieves 616 QPM, where the unified framework reduces TPOT by 36%, and the superimposition of OmniProxy further slashes TTFT by 38%. The project is open-sourced at [this https URL](https://gitee.com/omniai/omniinfer).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 45 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

38 of 45 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scale15 Aug
  3. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool25 Jun
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  5. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  6. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
18 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2025Kimi K2: Open Agentic Intelligencecited by 567
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  8. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  9. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  10. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  11. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  12. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 56
  13. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  14. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMscited by 27
  15. 2025Serving Large Language Models on Huawei CloudMatrix384cited by 24
  16. 2024Ada-K Routing: Boosting the Efficiency of MoE-based LLMscited by 16
  17. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  18. 2025A Survey of LLM Inference Systemscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation
  2. 2026AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall ↗
  3. 2026Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
  4. 2025ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput
  5. 2026AE-LLM: Adaptive Efficiency Optimization for Large Language Models
  6. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving
  7. 2026How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
  8. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure