FlashBoot: Sub-Second Weight Loading for Large Models at Rack Scale
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Flagship Mixture-of-Experts (MoE) models are growing fast along two axes at once: total parameter count and the number of experts. In elastic deployment scenarios, many GPUs across many nodes must become serving-ready quickly, and this growth makes weight loading a noticeable part of the latency budget. Even on NVIDIA's GB300 NVL72, today's state-of-the-art loaders leave most of that bandwidth unused. The losses are structural: (C1) weight memory is fragmented into tens of thousands of per-tensor objects, so transfers run far below link bandwidth; (C2) cross-node replication is gated by NCCL communicator setup, which costs 10-110 s before a single weight byte moves; and (C3) the existing cross-node GPU->GPU clone path is serial and scales poorly to concurrent multi-node bring-up. We present FlashBoot, a hardware-friendly, framework-workflow co-designed weight-loading subsystem built on SGLang. At its core is FabricArena, a contiguous, exportable and inter-node addressable tensor memory layout. On top of it, FlashLoad loads from CPU as a single bulk, zero-copy transfer, and FlashClone replicates a resident model from a remote GPU via a remote-mapping mechanism that removes NCCL setup. In experiments on NVL72 with DeepSeek-V4-Pro and DeepSeek-V4-Flash, FlashClone maps remote weight memory in ~10 ms (versus 10-110 s for NCCL) and sustains >=700 GB/s per clone. Against the state of the art, FlashBoot accelerates single-node weight loading by up to 50x (from 20.1 s to 0.4 s) and concurrent rack-level weight loading by >270x (from 87 s to 0.32 s). Our code will be made publicly available.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. First posted on arXiv 9 Aug 2026.
Builds on
No reference list yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025FlashMoE: Fast Distributed MoE in a Single Kernel
- 2025Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference
- 2026FEPLB: Exploiting Copy Engines for Nearly Free MoE Load Balancing in Distributed Training
- 2026CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
- 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
- 2026Mixture-of-Kittens: MoE Megakernel for NVL72s
- 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
- 2026MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs