Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
Not on the map. Why?
Abstract (arXiv)
Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47× and P99 TPOT by up to 3.40×, improving decode throughput by up to 2.1×, and reducing energy per token by up to 48.6%. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 52 references that are in the map. First posted on arXiv 8 Apr 2026.
Builds on
18 of 52 references.
References · 18
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
- 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
- 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
- 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
6 more
- 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
- 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domainscited by 14
- 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-locationcited by 10
- 2025Niyama : Breaking the Silos of LLM Inference Servingcited by 6
- 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Servingcited by 5
- 2023Lovelock: Towards Smart NIC-hosted Clusterscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Efficient LLM inference solution on Intel GPU
- 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
- 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
- 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
- 2025SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
- 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
- 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference
- 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching