# Parallax: Efficient LLM Inference Service over Decentralized Environment

arXiv: https://arxiv.org/abs/2509.26182 · HTML: https://mlsanity.com/paper/2509.26182/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 25 of 25)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism](https://mlsanity.com/paper/2404.09526/) (2024)
- [Decentralized Training of Foundation Models in Heterogeneous Environments](https://mlsanity.com/paper/2206.01288/) (2022)
- [Petals: Collaborative Inference and Fine-tuning of Large Models](https://mlsanity.com/paper/2209.01188/) (2022)
- [SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient](https://mlsanity.com/paper/2301.11913/) (2023)
- [Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow](https://mlsanity.com/paper/2406.01566/) (2024)
- [Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelism](https://mlsanity.com/paper/2211.13878/) (2022)
- [HexGen: Generative Inference of Large Language Model over Heterogeneous Environment](https://mlsanity.com/paper/2311.11514/) (2023)
- [Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees](https://mlsanity.com/paper/2206.01299/) (2022)
- [ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments](https://mlsanity.com/paper/2502.09334/) (2025)
- [Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs](https://mlsanity.com/paper/2502.00722/) (2025)
- [HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment](https://mlsanity.com/paper/2502.07903/) (2025)
- [Improving Automatic Parallel Training via Balanced Memory Workload Optimization](https://mlsanity.com/paper/2307.02031/) (2023)
- [LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind](https://mlsanity.com/paper/2508.15601/) (2025)
- [HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware](https://mlsanity.com/paper/2409.01143/) (2024)
- [DeServe: Towards Affordable Offline LLM Inference via Decentralization](https://mlsanity.com/paper/2501.14784/) (2025)
- [SAKSHI: Decentralized AI Platforms](https://mlsanity.com/paper/2307.16562/) (2023)
- [Towards Decentralized and Sustainable Foundation Model Training with the Edge](https://mlsanity.com/paper/2507.01803/) (2025)
- [PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Serving](https://mlsanity.com/paper/2504.20101/) (2025)

## Built on by (most cited first, 12 of 12)

- [AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs](https://mlsanity.com/paper/2511.00796/) (2025)
- [BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimization](https://mlsanity.com/paper/2602.10729/) (2026)
- [OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration](https://mlsanity.com/paper/2602.12151/) (2026)
- [Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets](https://mlsanity.com/paper/2608.19147/) (2026)
- [ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning](https://mlsanity.com/paper/2602.02192/) (2026)
- [Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput](https://mlsanity.com/paper/2511.11733/) (2025)
- [Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamics](https://mlsanity.com/paper/2604.07144/) (2026)
- [Trust-Aware Routing for Distributed Generative AI Inference at the Edge](https://mlsanity.com/paper/2603.28622/) (2026)
- [Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization](https://mlsanity.com/paper/2604.21072/) (2026)
- [HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Scheduling](https://mlsanity.com/paper/2605.16637/) (2026)
- [Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines](https://mlsanity.com/paper/2609.34045/) (2026)
- [Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure](https://mlsanity.com/paper/2609.38697/) (2026)

## Abstract (arXiv)

Deploying a large language model (LLM) inference service remains costly because centralized serving depends on specialized GPU clusters and high-bandwidth interconnects in datacenters. An appealing alternative is to leverage collaborative decentralized GPU pools. However, heterogeneity in GPU and limited interconnected network bandwidth, along with potentially dynamic availability, make efficient scheduling the central challenge in this scenario. In this paper, we present Parallax, a decentralized LLM serving system that turns a pool of heterogeneous GPUs into an efficient inference platform via a two-phase scheduler. Parallax decomposes planning into (i) model allocation, which places layers of each replica across diverse GPUs to jointly optimize latency and throughput under memory and link-bandwidth constraints, and (ii) request-time GPU pipeline selection, which stitches layers from different replicas into end-to-end execution chains that balance load and adapt to current conditions. We implement Parallax and evaluate it on open-source LLMs deployed over real volunteer nodes. Parallax consistently reduces latency and increases throughput relative to decentralized baselines, demonstrating that principled scheduling can make volunteer compute a practical, affordable substrate for LLM inference. Github Repo at: https://github.com/GradientHQ/parallax.