mlsanityData to 14 Oct 2026

LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization

arXiv 2608.21836 ↗First posted 22 Aug 2026cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Large language models have become increasingly capable agents for low-level code and kernel optimization, but isolated kernel benchmarks provide only a proxy for the deployment behavior that matters in language-model inference. We identify a benchmark-to-deployment gap: candidate kernels that appear correct and fast in standalone harnesses can exhibit different performance, safety, or phase behavior after integration into a real inference workload. We introduce LLM4LLM, a deployment-aware closed-loop optimization framework that starts from a target inference script, extracts phase-aware optimization tasks, searches with an experience-guided episodic agent, and accepts patches through in-model validation. Across ten language-model inference workloads on A100 and H100 GPUs, LLM4LLM improves end-to-end latency for every evaluated model, achieving 3.91×/6.98× geometric-mean speedups on A100/H100; as supporting kernel-level evidence, it also attains up to 2.745× GeoMean speedup on KernelBench Level 2.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 39 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

28 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026KernelSkill: A Multi-Agent Framework for GPU Kernel Optimization10 Mar
  2. 2026KernelBlaster: Continual Cross-Task CUDA Optimization via Memory-Augmented In-Context Reinforcement Learning15 Feb
  3. 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization16 Sep

Other references · 25

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2023Teaching Large Language Models to Self-Debugcited by 988
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
13 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  2. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  5. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
  6. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 201
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  8. 2018Learning to Optimize Tensor Programscited by 140
  9. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  10. 2018Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractionscited by 85
  11. 2025CUDA-L1: Improving CUDA Optimization via Contrastive Reinforcement Learningcited by 61
  12. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
  13. 2022TensorIR: An Abstraction for Automatic Tensorized Program Optimizationcited by 38

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulator
  2. 2025STARK: Strategic Team of Agents for Refining Kernels
  3. 2024GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments
  4. 2026EGG: An Expert-Guided Agent Framework for Kernel Generation
  5. 2026KernelGenBench: Can LLMs and Agents Write Efficient Kernels Across Operator Sources and Hardware Platforms?
  6. 2024Inference Performance Optimization for Large Language Models on CPUs
  7. 2026FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems
  8. 2026KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware