Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization
On the map: Efficient LLM inference and serving → Speculative decoding
Abstract (arXiv)
Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional centralized inference. However, the low cross-node network bandwidth makes communication the primary bottleneck. In this paper, we introduce BloomBee, an internet-scale distributed LLM inference framework. BloomBee integrates LLM-layer assignment, micro-batching and tensor offloading to optimize communication from multiple dimensions. Additionally, BloomBee formulates the coordination of these techniques as an optimization problem and solves it using dynamic programming. BloomBee also customizes lossless compression and speculative decoding according to low-bandwidth network settings to reduce communication overhead. We evaluate BloomBee across a spectrum of network environments and show that it improves service throughput by up to 1.76x. It also reduces average latency by up to 43.20% compared to state-of-the-art decentralized LLM inference systems. BloomBee is open-sourced.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 44 references that are in the map. First posted on arXiv 22 Apr 2026.
Builds on
16 of 44 references.
Closest ancestors · 2
Same direction, newest first
- 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
- 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
Other references · 14
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
- 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
- 2021Distributed Deep Learning in Open Collaborationscited by 58
- 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
- 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
- 2023HexGen: Generative Inference of Large Language Model over Heterogeneous Environmentcited by 31
- 2025Auto-ARGUE: LLM-Based Report Generation Evaluationcited by 14
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission
- 2025Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
- 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
- 2025Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference
- 2025DeServe: Towards Affordable Offline LLM Inference via Decentralization
- 2026WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
- 2026Towards Distributed Inference of LLMs on a P2P Network
- 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗