mlsanityData to 7 Oct 2026

SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions

arXiv 2606.16332 ↗First posted 15 Jun 2026cs.DC cs.AI cs.PFcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide high-throughput matrix execution within the CPU. For LLM inference, however, these units are not a universal replacement for conventional CPU cores: prefill, decode, attention, and KV-cache operations expose different arithmetic intensities, vector behavior, and layout requirements, while SME units and CPU cores still compete for shared memory bandwidth. This paper studies this mismatch through a roofline-based characterization of SME-enabled CPUs and uses the resulting model to guide operator-level execution choices. We present SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each operator shape. SMEPilot partitions matrix work across SME and CPU cores at tile granularity, overlaps SME-suitable matrix stages with CPU-suitable vector stages in attention, and maintains layout state so packed tensor representations are reused rather than repeatedly rebuilt on critical paths. Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B on phone, PC, and server platforms, SMEPilot improves end-to-end inference performance by up to 3.94×.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 40 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

20 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Serving19 May
  2. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models3 Jun
  3. 2024Inference Acceleration for Large Language Models on CPUs4 Mar
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 15

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  8. 2024Large Language Models: A Surveycited by 506
  9. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  11. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  12. 2024Hello SME! Generating Fast Matrix Multiplication Kernels Using the Scalable Matrix Extensioncited by 5
3 more
  1. 2025SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUscited by 3
  2. 2025Demystifying ARM SME to Optimize General Matrix Multiplicationscited by 1
  3. 2026ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  2. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
  3. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV
  4. 2026BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators
  5. 2024TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
  6. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
  7. 2025MIST: A Co-Design Framework for Heterogeneous, Multi-Stage LLM Inference
  8. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs