mlsanityData to 14 Oct 2026

Think Before You Grid-Search: Floor-First Triage for LLM Serving

arXiv 2607.05876 ↗First posted 7 Jul 2026cs.PF cs.AI cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed. We argue for the reverse discipline: estimation is the analytical layer of profiling -- without it, optimization degenerates to grid search. Floor First is a residual-driven triage workflow. Each decode step is modeled as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity); summing within a resource and maximizing across resources gives an optimistic floor, the plain sum a pessimistic one. Where a measurement lands inside this [max, sum] interval reads out overlap quality before any profiler is opened, and profilers escalate only on residuals above a stated threshold. Deployment alternatives are compared by wall ordering -- which resource wall binds first as load grows -- rather than by point benchmarks. The account is compositional: new attention or state-space variants enter by declaring one module, and the workflow ships as a zero-dependency calculator plus an agent skill that enforces the discipline in agentic optimization loops. As a case study we analyze a DeepSeek-V3.2-style 671B MoE/MLA model on 16 NVIDIA H20 GPUs, whose ridge point of ~74 FLOP/byte (vs ~590 for H100) makes it an extreme decode-oriented part. The floors show TP16 decoding is KV-capacity-limited to ~70 concurrent 8K requests; sparse attention removes the KV-bandwidth term but not the capacity wall; an EP16+DP-attention layout accepts slightly worse same-batch weight traffic for an order-of-magnitude higher capacity wall (~644) -- while single-stream latency favors TP by 2.4x. The layout judgment is thus a computable function of the operating point, explaining why production deployments on identical hardware have shipped opposite attention layouts.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 54 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

38 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode28 May
  2. 2026How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving27 May
  3. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving9 Jan
  4. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  5. 2024Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models3 Jun
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insights26 Feb
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 30

  1. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  2. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  3. 2022Efficiently Scaling Transformer Inferencecited by 539
  4. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  5. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  6. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  7. 2025Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inferencecited by 33
  8. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoEcited by 19
  9. 2023A Hardware Evaluation Framework for Large Language Model Inferencecited by 14
  10. 2024LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Servicescited by 14
  11. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  12. 2025LIMINAL: Exploring The Frontiers of LLM Decode Performancecited by 11
18 more
  1. 2026KernelSkill: A Multi-Agent Framework for GPU Kernel Optimizationcited by 11
  2. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  3. 2024Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inferencecited by 9
  4. 2025Inference economics of language modelscited by 9
  5. 2026XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inferencecited by 7
  6. 2026MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inferencecited by 7
  7. 2025FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUscited by 5
  8. 2026RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysiscited by 5
  9. 2025Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Expertscited by 4
  10. 2026Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimizationcited by 4
  11. 2025ASAP: an Agentic Solution to Auto-optimize Performance of Large-Scale LLM Trainingcited by 3
  12. 2026From Human Guidance to Autonomy: Agent Skill System for End-to-End LLM Deployment on Spatial NPUscited by 2
  13. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulatorcited by 2
  14. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  15. 2026MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithmcited by 1
  16. 2026EPOCH: An Agentic Protocol for Multi-Round System Optimizationcited by 1
  17. 2026GQLA: Group-Query Latent Attention for Hardware-Adaptive Decodingcited by 1
  18. 2026Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principlescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  2. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
  3. 2026LAPS: A Length-Aware-Prefill LLM Serving System
  4. 2026FleetSieve: Decision-Critical Profiling for SLO-Aware LLM Fleet Configuration
  5. 2026Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces
  6. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling
  7. 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Serving
  8. 2026LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications