mlsanityData to 14 Oct 2026

RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems

arXiv 2609.12551 ↗First posted 11 Sep 2026cs.DC cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

AI is beginning to make substantive contributions to LLM inference optimization. Existing AI optimizations are predominantly profiling-based. Profiling-bound feedback confines the search to the capabilities and performance of an existing software stack, preventing a fundamentally better architecture of LLM inference systems from being identified. To enable the AI-driven LLM inference system architecting loop, we argue that a general workload representation, a verifiable mutation space, and an implementation-independent evaluator are required. We present the RoofLang domain-specific language (DSL) that provides these features. In our evaluation, RoofLang reveals that DeepSeek V4-series models could achieve 3.5-39.5× higher peak decode throughput than other representative models. This gap is disproportionate to their total parameter counts and arises largely from compact KV-cache designs that support larger batches and reduce memory traffic. A persistent optimizer agent further discovered several new architectures that improved both throughput and interactivity of DeepSeek V4 Pro on NVIDIA B300 by 6.23-50.1%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 39 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

19 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Frontier: Towards Comprehensive and Accurate LLM Inference Simulation20 May
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 16

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  5. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  8. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  9. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
  10. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  11. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  12. 2026FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systemscited by 25
4 more
  1. 2025GPU Kernel Scientist: An LLM-Driven Framework for Iterative Kernel Optimizationcited by 22
  2. 2026InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agentscited by 5
  3. 2026Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimizationcited by 4
  4. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulatorcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models
  2. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving
  3. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  4. 2023Efficient LLM Inference on CPUs
  5. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  6. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  7. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
  8. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMV