MoLink: Distributed and Efficient Serving Framework for Large Models
On the map: Efficient LLM inference and serving → Serving systems and request scheduling
Abstract (arXiv)
Large language models represent a groundbreaking shift in generative AI. Yet, these advances come with a significant challenge: the high cost of model serving. To mitigate these costs, consumer-grade GPUs emerge as a more affordable alternative. This presents an opportunity for more cost-efficient LLM serving by leveraging these GPUs. However, it is non-trivial to achieve high-efficiency LLM serving on consumer-grade GPUs, mainly due to two challenges: 1) these GPUs are often deployed in limited network conditions; 2) these GPUs often exhibit heterogeneity in host systems. To address these challenges, we present MoLink, a distributed LLM serving system for large models. It incorporates several key techniques, enabling efficient LLM serving on heterogeneous and weakly connected consumer-grade GPUs. Our experiments demonstrate that it achieves throughput improvements of up to 458% and cost-profit margin improvements of up to 151%, compared to state-of-the-art systems. MoLink allows users on Windows, Linux, and containerized VMs to seamlessly integrate GPUs with just a few lines of code over Ethernet or public networks. Currently, it supports 18 mainstream architectures of open-source large language models. The source code is publicly available https://github.com/oldcpple/MoLink.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 30 references that are in the map. First posted on arXiv 7 Jul 2025.
Builds on
16 of 30 references.
Closest ancestors · 2
Same direction, newest first
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
- 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
Other references · 14
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
- 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
- 2021Distributed Deep Learning in Open Collaborationscited by 58
- 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 58
- 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
- 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
- 2020Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Expertscited by 43
- 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization
- 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
- 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
- 2025DeServe: Towards Affordable Offline LLM Inference via Decentralization
- 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
- 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
- 2025Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity
- 2026Lodestar: An Online-Learning LLM Inference Router