mlsanityData to 14 Oct 2026

CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory

arXiv 2503.02354 ↗First posted 4 Mar 2025cs.DC cs.AI cs.PFcited by 5

Not on the map. Why?

Abstract (arXiv)

Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. The Collaboration-of-Experts (CoE) approach integrates multiple expert models, improving the accuracy of generated results and offering great potential for precision-critical applications, such as automatic circuit board quality inspection. However, deploying CoE serving systems presents challenges to memory capacity due to the large number of experts required, which can lead to significant performance overhead from frequent expert switching across different memory and storage tiers. We propose CoServe, an efficient CoE model serving system on heterogeneous CPU and GPU with limited memory. CoServe reduces unnecessary expert switching by leveraging expert dependency, a key property of CoE inference. CoServe introduces a dependency-aware request scheduler and dependency-aware expert management for efficient inference. It also introduces an offline profiler to automatically find optimal resource allocation on various processors and devices. In real-world intelligent manufacturing workloads, CoServe achieves 4.5× to 12× higher throughput compared to state-of-the-art systems.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

25 of 41 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2024Mixtral of Expertscited by 1,823
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  11. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  12. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
13 more
  1. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 109
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  3. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  4. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 91
  5. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  6. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  7. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  8. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  9. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  10. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  11. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  12. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  13. 2024Bench-CoE: a Framework for Collaboration of Experts from Benchmarkcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inferencecited by 13
  2. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  3. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  4. 2025CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answeringcited by 0
  5. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0