mlsanityData to 7 Oct 2026

A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference

arXiv 2511.15950 ↗First posted 20 Nov 2025cs.DC cs.AI cs.ARcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

A vertically integrated, end-to-end, research prototype system combines 288 NorthPole neural inference accelerator cards, offline training algorithms, a high-performance runtime stack, and a containerized inference pipeline to deliver a scalable and efficient cloud inference service. The system delivers 115 peta-ops at 4-bit integer precision and 3.7 PB/s of memory bandwidth across 18 2U servers, while consuming only 30 kW of power and weighing 730 kg in a 0.67 m^2 42U rack footprint. The system can run 3 simultaneous instances of the 8-billion-parameter open-source IBM Granite-3.3-8b-instruct model at 2,048 context length with 28 simultaneous users and a per-user inter-token latency of 2.8 ms. The system is scalable, modular, and reconfigurable, supporting various model sizes and context lengths, and is ideal for deploying agentic workflows for enterprise AI applications in existing data center (cloud, on-prem) environments. For example, the system can support 18 instances of a 3-billion-parameter model or a single instance of a 70-billion-parameter model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 6 of its 20 references that are in the map. First posted on arXiv 20 Nov 2025.

Builds on

6 of 20 references.

References · 6

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2025Small Language Models are the Future of Agentic AIcited by 254
  4. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  5. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  6. 2025SiLQ: Simple Large Language Model Quantization-Aware Trainingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
  2. 2026FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration
  3. 2025LoopLynx: A Scalable Dataflow Architecture for Efficient LLM Inference
  4. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
  5. 2026BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators
  6. 2025Understanding the Performance and Power of LLM Inferencing on Edge Accelerators
  7. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  8. 2024AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality