mlsanityData to 14 Oct 2026

EPD-Serve: A Flexible Multimodal EPD Disaggregation Inference Serving System On Ascend

arXiv 2601.11590 ↗First posted 5 Jan 2026cs.DC cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

With the widespread adoption of large multimodal models, efficient inference across text, image, audio, and video modalities has become critical. However, existing multimodal inference systems typically employ monolithic architectures that tightly couple the Encode, Prefill, and Decode stages on homogeneous hardware, neglecting the heterogeneous computational characteristics of each stage. This design leads to inefficient resource utilization and limited system throughput. To address these issues, we propose EPD-Serve, a stage-level disaggregated inference serving system for multimodal models. EPD-Serve decouples the inference pipeline into independent Encode, Prefill, and Decode stages, enabling logical isolation and flexible co-located deployment through dynamic orchestration. Leveraging the Ascend interconnect topology, EPD-Serve introduces asynchronous feature prefetching between Encode and Prefill stages and a hierarchical grouped KV cache transmission mechanism between Prefill and Decode stages to improve cross-node communication efficiency. In addition, EPD-Serve incorporates multi-route scheduling, instance-level load balancing, and multi-stage hardware co-location with spatial multiplexing to better support diverse multimodal workloads. Comprehensive experiments on multimodal understanding models demonstrate that, under high-concurrency scenarios, EPD-Serve improves end-to-end throughput by 57.37-69.48% compared to PD-disaggregated deployment, while satisfying strict SLO constraints, including TTFT below 2000 ms and TPOT below 50 ms. These results highlight the effectiveness of stage-level disaggregation for optimizing multimodal large model inference systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 16 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

13 of 16 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving2 Feb

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  4. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  7. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  8. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  9. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 97
  10. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 68
  11. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 35
  12. 2025HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Servingcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregation
  2. 2026Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference
  3. 2026EAServe: Encode-Aware Disaggregated Serving for Multimodal Large Language Models
  4. 2025Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devices
  5. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
  6. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
  7. 2026Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling
  8. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling