mlsanityData to 7 Oct 2026

Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets

arXiv 2608.19147 ↗First posted 19 Aug 2026cs.DC cs.AI cs.SEcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of AIPCs, working together over an ordinary network, can serve models beyond the capability of any single one. We use pipeline parallelism: a model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so that every machine runs one shard and passes activations to the next. Three techniques make this fast enough to be useful. First, we recover the speed of the unsplit model: a naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization, and injecting a beam_idx Gather into each shard triggers that optimization (the IndirectKVCache fusion) and brings the shards to parity. Second, we leverage speculative decoding on stateful OpenVINO models. Third, the pipeline serves several users at once by interleaving their requests across the stages, each request carrying its own cache (micro-batching). Together, a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput of the unsplit model on the same hardware, and the gap widens under simulated wide-area latency. The same design scales to a 70B model that no single fleet member can hold: a four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed, with output token-for-token identical to the same four-node pipeline decoding without speculation. Code, raw benchmark logs, and reproduction scripts ship as a self-contained package at https://github.com/labscommunity/pipeline-sharded-inference-paper (in the top-level reproduction/ directory).

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 21 references that are in the map. First posted on arXiv 19 Aug 2026.

Builds on

13 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment30 Sep

Other references · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  6. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  7. 2026Gemma 4 Technical Reportcited by 187
  8. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  9. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  10. 2025Model-Distributed Inference for Large Language Models at the Edgecited by 8
  11. 2025TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inferencecited by 8
  12. 2025SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimizationcited by 0
  2. 2026Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  2. 2025Learning to Shard: RL for Co-optimizing the Parallelism Degrees and Per-operator Sharding Dimensions in Distributed LLM Inference
  3. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  4. 2026PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
  5. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  6. 2025SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding
  7. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  8. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study