mlsanityData to 7 Oct 2026

Meeting SLOs, Slashing Hours: Automated Enterprise LLM Optimization with OptiKIT

arXiv 2601.20408 ↗First posted 28 Jan 2026cs.DC cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Enterprise LLM deployment faces a critical scalability challenge: organizations must optimize models systematically to scale AI initiatives within constrained compute budgets, yet the specialized expertise required for manual optimization remains a niche and scarce skillset. This challenge is particularly evident in managing GPU utilization across heterogeneous infrastructure while enabling teams with diverse workloads and limited LLM optimization experience to deploy models efficiently. We present OPTIKIT, a distributed LLM optimization framework that democratizes model compression and tuning by automating complex optimization workflows for non-expert teams. OPTIKIT provides dynamic resource allocation, staged pipeline execution with automatic cleanup, and seamless enterprise integration. In production, it delivers more than 2x GPU throughput improvement while empowering application teams to achieve consistent performance improvements without deep LLM optimization expertise. We share both the platform design and key engineering insights into resource management, pipeline orchestration, and integration patterns that enable large-scale, production-grade democratization of model optimization. Finally, we open-source the system to enable external contributions and broader reproducibility.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

28 of 41 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  12. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
16 more
  1. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  2. 2023A Survey on Model Compression for Large Language Modelscited by 263
  3. 2021Accelerating Sparse Deep Neural Networkscited by 213
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2022FP8 Formats for Deep Learningcited by 187
  6. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 143
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  8. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  9. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  10. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  11. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 20
  12. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  13. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  14. 2024Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMscited by 11
  15. 2024SCOOT: SLO-Oriented Performance Tuning for LLM Inference Enginescited by 9
  16. 2025High-Throughput LLM inference on Heterogeneous Clusterscited by 2

Built on by

Not cited yet.