mlsanityData to 7 Oct 2026

TCM-Serve: Modality-aware Scheduling for Multimodal Large Language Model Inference

arXiv 2603.26498 ↗First posted 27 Mar 2026cs.DC cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) power platforms like ChatGPT, Gemini, and Copilot, enabling richer interactions with text, images, and videos. These heterogeneous workloads introduce additional inference stages, such as vision preprocessing and encoding, that inflate latency and memory demand. Existing LLM serving systems, optimized for text-only workloads, fail under multimodality: large requests (e.g., videos) monopolize resources, causing severe head-of-line blocking and performance degradation. Our key insight is that multimodal requests differ by orders of magnitude in resource demands, which we capture through a simple abstraction: videos behave like trucks, images like cars, and text like motorcycles. We design TCM-Serve, a modality-aware scheduler that lets motorcycles flow quickly through cars and trucks, ensuring interactive responsiveness while avoiding starvation. TCM-Serve classifies requests, prioritizes them dynamically, and applies aging to avoid starvation. Evaluation across state-of-the-art MLLMs shows that TCM-Serve reduces, on average, time-to-first-token (TTFT) by 54% overall, and by 78.5% for latency-critical requests, compared to current systems. TCM-Serve delivers LLM-like responsiveness for MLLMs, with modality-aware scheduling and by making the most efficient use of the available resources.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 41 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

24 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving2 Feb
  2. 2024Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU11 Sep
  3. 2024Efficient Inference of Vision Instruction-Following Models with Elastic Cache25 Jul
  4. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference9 May

Other references · 20

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  7. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  10. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  11. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  12. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
8 more
  1. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  2. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  3. 2024Efficient LLM Scheduling by Learning to Rankcited by 83
  4. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  5. 2023Fairness in Serving Large Language Modelscited by 77
  6. 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancingcited by 7
  7. 2025SLO-Aware Scheduling for Large Language Model Inferencescited by 7
  8. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Servingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
  2. 2025Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference ↗
  3. 2026Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
  4. 2026Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines
  5. 2025Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency
  6. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha
  7. 2025Enabling Disaggregated Multi-Stage MLLM Inference via GPU-Internal Scheduling and Resource Sharing
  8. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference