mlsanityData to 14 Oct 2026

Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving

arXiv 2507.18454 ↗First posted 19 May 2025cs.AR cs.AI cs.DCcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

CPUs are critical for LLM serving due to their availability, cost efficiency, and edge applicability. However, efficient CPU serving is hindered by conflicting prefill/decode resource demands under non-disaggregated deployment constraints--existing solutions fail to avoid cross-phase interference, ignore sub-NUMA hardware structures, and deliver suboptimal dynamic-shape kernel performance. We propose Sandwich, a full-stack CPU LLM serving system with three core innovations addressing these challenges: (1) seamless phase-wise plan switching to eliminate cross-phase interference; (2) TopoTree, a tree-based hardware abstraction for automated substructure-aware (e.g., LLC slices) partial core allocation; (3) fast-start-then-finetune dynamic-shape tensor program generation. Across five x86/ARM CPU platforms, Sandwich achieves an average 2.01x end-to-end speedup and up to 3.40x latency reduction over state-of-the-art systems. Its kernels match static compiler performance with three orders of magnitude lower tuning cost.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 43 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

11 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization2 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 9

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 201
  6. 2018Learning to Optimize Tensor Programscited by 140
  7. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  8. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  9. 2023CASSINI: Network-Aware Job Scheduling in Machine Learning Clusterscited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  2. 2026SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS
  2. 2024Tackling the Dynamicity in a Production LLM Serving System with SOTA Optimizations via Hybrid Prefill/Decode/Verify Scheduling on Efficient Meta-kernels
  3. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing
  4. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  5. 2026Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
  6. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructure
  7. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
  8. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure