mlsanityData to 14 Oct 2026

Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters

arXiv 2504.08791 ↗First posted 7 Apr 2025cs.DC cs.AIcited by 3

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

On-device inference offers privacy, offline use, and instant response, but consumer hardware restricts large language models (LLMs) to low throughput and capability. To overcome this challenge, we present prima.cpp, a distributed on-device inference system that runs 30-70B LLMs on consumer home clusters with mixed CPUs/GPUs, insufficient RAM/VRAM, slow disks, Wi-Fi links, and heterogeneous OSs. We introduce pipelined-ring parallelism (PRP) to overlap disk I/O with compute and communication, and address the prefetch-release conflict in mmap-based offloading. We further propose Halda, a heterogeneity-aware scheduler that co-optimizes per-device CPU/GPU workloads and device selection under RAM/VRAM constraints. On four consumer home devices, a 70B model reaches 674 ms/token TPOT with <6% memory pressure, and a 32B model with speculative decoding achieves 26 tokens/s. Compared with llama.cpp, exo, and dllama, our proposed prima.cpp achieves 5-17 times lower TPOT, supports fine-grained model sizes from 8B to 70B, ensures broader cross-OS and quantization compatibility, and remains OOM-free, while also being Wi-Fi tolerant, privacy-preserving, and hardware-independent. The code is available at https://github.com/OpenCPIL/prima.cpp.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 7 of its 18 references that are in the map. First posted on arXiv 7 Apr 2025.

Builds on

7 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 6

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2021Embodied Intelligence via Learning and Evolutioncited by 163
  3. 2015Parallelizing the dual revised simplex methodcited by 89
  4. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  5. 2023LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devicescited by 3
  6. 2025Distributed On-Device LLM Inference With Over-the-Air Computationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktopscited by 0
  2. 2026Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructurecited by 0
  3. 2026Cascadia: Resident 975B MoE Inference on Eleven AI PCscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LOIP:Collaborative Lossless LLM Inference Serving with Offloading-based Pipeline Parallelism on Edge Devices
  2. 2026LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
  3. 2025Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism
  4. 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
  5. 2026When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference
  6. 2026Federated Inference for Heterogeneous LLM Communication and Collaboration
  7. 2026Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
  8. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices