mlsanityData to 14 Oct 2026

Parallax: Efficient LLM Inference Service over Decentralized Environment

arXiv 2509.26182 ↗First posted 30 Sep 2025cs.DCcited by 12

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Deploying a large language model (LLM) inference service remains costly because centralized serving depends on specialized GPU clusters and high-bandwidth interconnects in datacenters. An appealing alternative is to leverage collaborative decentralized GPU pools. However, heterogeneity in GPU and limited interconnected network bandwidth, along with potentially dynamic availability, make efficient scheduling the central challenge in this scenario. In this paper, we present Parallax, a decentralized LLM serving system that turns a pool of heterogeneous GPUs into an efficient inference platform via a two-phase scheduler. Parallax decomposes planning into (i) model allocation, which places layers of each replica across diverse GPUs to jointly optimize latency and throughput under memory and link-bandwidth constraints, and (ii) request-time GPU pipeline selection, which stitches layers from different replicas into end-to-end execution chains that balance load and adapt to current conditions. We implement Parallax and evaluate it on open-source LLMs deployed over real volunteer nodes. Parallax consistently reduces latency and increases throughput relative to decentralized baselines, demonstrating that principled scheduling can make volunteer compute a practical, affordable substrate for LLM inference. Github Repo at: https://github.com/GradientHQ/parallax.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 32 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

25 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind21 Aug
  2. 2025PlanetServe: A Decentralized, Scalable, and Privacy-Preserving Overlay for Democratizing Large Language Model Serving27 Apr
  3. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments13 Feb
  4. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs2 Feb
  5. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr

Other references · 20

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  8. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  9. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  10. 2023SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficientcited by 59
  11. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  12. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 53
8 more
  1. 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
  2. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 30
  3. 2025HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environmentcited by 20
  4. 2023Improving Automatic Parallel Training via Balanced Memory Workload Optimizationcited by 14
  5. 2024HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardwarecited by 7
  6. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  7. 2023SAKSHI: Decentralized AI Platformscited by 3
  8. 2025Towards Decentralized and Sustainable Foundation Model Training with the Edgecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 10.

12 papers.

  1. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  2. 2026BOute: Cost-Efficient LLM Serving with Heterogeneous LLMs and GPUs via Multi-Objective Bayesian Optimizationcited by 6
  3. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 3
  4. 2026Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleetscited by 3
  5. 2026ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learningcited by 2
  6. 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughputcited by 1
  7. 2026Autopoiesis: A Self-Evolving System Paradigm for LLM Serving Under Runtime Dynamicscited by 1
  8. 2026Trust-Aware Routing for Distributed Generative AI Inference at the Edgecited by 0
  9. 2026Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimizationcited by 0
  10. 2026HexAGenT: Efficient Agentic LLM Serving via Workflow- and Heterogeneity-Aware Schedulingcited by 0
  11. 2026Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machinescited by 0
  12. 2026Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
  2. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  3. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  4. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  5. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  6. 2024KVDirect: Distributed Disaggregated LLM Inference
  7. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  8. 2026Lodestar: An Online-Learning LLM Inference Router