mlsanityData to 7 Oct 2026

KernelSight-LM: A Kernel-Level LLM Inference Simulator

arXiv 2606.28565 ↗First posted 26 Jun 2026cs.PF cs.AI cs.ARcited by 2

Not on the map. Why?

Abstract (arXiv)

As large language models (LLMs) move into production serving, practitioners must rapidly evaluate inference performance across diverse hardware, models, and serving parameters to meet cost and latency targets. However, the end-to-end behavior of LLMs couples serving-layer policies with low-level GPU kernel execution and rapidly evolving architectures, forcing slow, deployment-specific benchmarking that is hard to generalize. We present KernelSight-LM, a fine-grained inference simulator that models token-level execution and produces kernel-level latency breakdowns. It decomposes each serving step into a roofline kernel model with a learned efficiency term, a communication model, and a host-overhead model, composed through a discrete-event scheduler that also captures mechanisms like prefix caching and continuous batching. KernelSight-LM offers two prediction tiers that trade target-GPU data for accuracy. The cross-generation tier uses no target-GPU measurements, only hardware specifications and kernel microbenchmarks from previously profiled GPUs, and predicts per-kernel latency on an unseen GPU generation to 12.1% error, a 1.8x improvement over the roofline baseline (22.0%). A second target-measured tier adds one model-agnostic kernel-microbenchmark sweep on the target GPU, sharpening per-kernel error to 3.8%, a 7.3x improvement over a comparable baseline (27.7%). Both tiers require far less target-GPU data than the prior systems they extend. In our simulator, these predictions yield end-to-end median (p50) errors across six model families of 15.4%, 12.8%, and 3.0% (TTFT, TPOT, throughput) in the cross-generation tier and 14.3%, 6.2%, and 2.7% in the target-measured tier, matching dedicated profiling tools while collecting far less on-device data. Beyond prediction, its kernel-level bottleneck breakdowns support hardware/software co-design and capacity planning.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 49 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

28 of 49 references.

References · 28

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  11. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
16 more
  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  3. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  7. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  8. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  9. 2021Torch.fx: Practical Program Capture and Transformation for Deep Learning in Pythoncited by 38
  10. 2018Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarkingcited by 35
  11. 2021A Runtime-Based Computational Performance Predictor for Deep Neural Network Trainingcited by 29
  12. 2024Forecasting GPU Performance for Deep Learning Training and Inferencecited by 15
  13. 2024APEX: An Extensible and Dynamism-Aware Simulator for Automated Parallel Execution in LLM Servingcited by 15
  14. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Servingcited by 9
  15. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systemscited by 6
  16. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructurecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Think Before You Grid-Search: Floor-First Triage for LLM Servingcited by 0
  2. 2026RoofLang: Enabling AI-Driven Architecting of LLM Inference Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
  2. 2023Efficient LLM inference solution on Intel GPU
  3. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  4. 2026LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization
  5. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  6. 2024Inference Performance Optimization for Large Language Models on CPUs
  7. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  8. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems