mlsanityData to 7 Oct 2026

AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

arXiv 2601.06288 ↗First posted 9 Jan 2026cs.LG cs.AI cs.DCcited by 9

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 10 of its 23 references that are in the map. First posted on arXiv 9 Jan 2026.

Builds on

10 of 23 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Serving26 Nov
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar

Other references · 7

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  7. 2022Galvatron: Efficient Transformer Training over Multiple GPUs Using Automatic Parallelismcited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.

  1. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulationcited by 4
  2. 2026How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Servingcited by 2
  3. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulatorcited by 2
  4. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inferencecited by 1
  5. 2026The Price of Anarchy in Disaggregated Inferencecited by 1
  6. 2026LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Samplingcited by 0
  7. 2026Rethinking LLMOps for Fraud and AML: Building a Compliance-Grade LLM Serving Stackcited by 0
  8. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0
  9. 2026Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  2. 2024Inference Performance Optimization for Large Language Models on CPUs
  3. 2026ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
  4. 2026RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
  5. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  6. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  7. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  8. 2026From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving