mlsanityData to 7 Oct 2026

ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving

arXiv 2502.00937 ↗First posted 2 Feb 2025cs.DC cs.AIcited by 18

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large multimodal models (LMMs) demonstrate impressive capabilities in understanding images, videos, and audio beyond text. However, efficiently serving LMMs in production environments poses significant challenges due to their complex architectures and heterogeneous characteristics across their multi-stage inference pipelines. We present the first comprehensive systems analysis of two prominent LMM architectures, decoder-only and cross-attention, across six representative open-source models, revealing key systems design implications. We also present an in-depth analysis of production LMM inference traces, uncovering unique workload characteristics, including variable, heavy-tailed request distributions and bursty traffic patterns. Based on these insights, we propose ModServe, a modular LMM serving system that decouples stages for independent optimization and adaptive scaling. ModServe dynamically reconfigures stages and handles bursty traffic with modality-aware scheduling and autoscaling to meet tail latency SLOs while minimizing costs. ModServe achieves 3.3-5.5x higher throughput (leading to 25-41.3% cost saving) while meeting SLOs on a 128-GPU cluster with production traces.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 55 references that are in the map. First posted on arXiv 2 Feb 2025.

Builds on

31 of 55 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification1 Dec
  2. 2024Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU11 Sep
  3. 2024Efficient Inference of Vision Instruction-Following Models with Elastic Cache25 Jul
  4. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
15 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  5. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  6. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  7. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  8. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 118
  9. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  10. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  11. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  12. 2024Queue management for slo-oriented large language model servingcited by 36
  13. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platformscited by 28
  14. 2023OptScaler: A Collaborative Framework for Robust Autoscaling in the Cloudcited by 2
  15. 2024Characterizing and Efficiently Accelerating Multimodal Generation Model Inferencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 13.

18 papers.

Sort:

By area:

  1. 2026Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scalecited by 9
  2. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  3. 2025Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Servingcited by 4
  4. 2026M*: A Modular, Extensible, Serving System for Multimodal Modelscited by 3
  5. 2025Chat with AI: The Surprising Turn of Real-time Video Communication from Human to AIcited by 2
  6. 2025Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharingcited by 1
  7. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristicscited by 1
  8. 2026FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applicationscited by 1
  9. 2025LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Modelscited by 0
  10. 2025From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Modelscited by 0
  11. 2026EPD-Serve: A Flexible Multimodal EPD Disaggregation Inference Serving System On Ascendcited by 0
  12. 2026QuickGrasp: Responsive Video-Language Querying Service via Accelerated Tokenization and Edge-Augmented Inferencecited by 0
6 more
  1. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneitycited by 0
  2. 2026TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inferencecited by 0
  3. 2026Time, Causality, and Observability Failures in Distributed AI Inference Systemscited by 0
  4. 2026LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?cited by 0
  5. 2026Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generationcited by 0
  6. 2026SARA: SLO-Aware Resource Allocation for Disaggregated Agentic LLM Servicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
  2. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregation
  3. 2025RServe: Overlapping Encoding and Prefill for Efficient LMM Inference
  4. 2025DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline
  5. 2025Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference ↗
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  7. 2026Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference
  8. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models