# Think Before You Grid-Search: Floor-First Triage for LLM Serving

arXiv: https://arxiv.org/abs/2607.05876 · HTML: https://mlsanity.com/paper/2607.05876/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 38 of its 54 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 30 of 38)

- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding](https://mlsanity.com/paper/2507.19427/) (2025)
- [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](https://mlsanity.com/paper/2505.09343/) (2025)
- [Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference](https://mlsanity.com/paper/2503.08311/) (2025)
- [MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE](https://mlsanity.com/paper/2505.19645/) (2025)
- [A Hardware Evaluation Framework for Large Language Model Inference](https://mlsanity.com/paper/2312.03134/) (2023)
- [LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services](https://mlsanity.com/paper/2410.02425/) (2024)
- [Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models](https://mlsanity.com/paper/2406.01698/) (2024)
- [MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems](https://mlsanity.com/paper/2505.11415/) (2025)
- [LIMINAL: Exploring The Frontiers of LLM Decode Performance](https://mlsanity.com/paper/2507.14397/) (2025)
- [KernelSkill: A Multi-Agent Framework for GPU Kernel Optimization](https://mlsanity.com/paper/2603.10085/) (2026)
- [Hardware-Efficient Attention for Fast Decoding](https://mlsanity.com/paper/2505.21487/) (2025)
- [Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference](https://mlsanity.com/paper/2407.14645/) (2024)
- [Inference economics of language models](https://mlsanity.com/paper/2506.04645/) (2025)
- [AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving](https://mlsanity.com/paper/2601.06288/) (2026)
- [XShare: Collaborative in-Batch Expert Sharing for Faster MoE Inference](https://mlsanity.com/paper/2602.07265/) (2026)
- [MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference](https://mlsanity.com/paper/2605.07363/) (2026)
- [FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUs](https://mlsanity.com/paper/2506.01969/) (2025)
- [RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis](https://mlsanity.com/paper/2602.11506/) (2026)
- [Rethinking LLM Inference Bottlenecks: Insights from Latent Attention and Mixture-of-Experts](https://mlsanity.com/paper/2507.15465/) (2025)
- [Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode](https://mlsanity.com/paper/2605.30571/) (2026)
- [Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization](https://mlsanity.com/paper/2606.26453/) (2026)
- [ASAP: an Agentic Solution to Auto-optimize Performance of Large-Scale LLM Training](https://mlsanity.com/paper/2511.03844/) (2025)

## Abstract (arXiv)

LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed. We argue for the reverse discipline: estimation is the analytical layer of profiling -- without it, optimization degenerates to grid search. Floor First is a residual-driven triage workflow. Each decode step is modeled as a five-dimensional resource vector (HBM bytes, FLOPs, network bytes, network messages, KV capacity); summing within a resource and maximizing across resources gives an optimistic floor, the plain sum a pessimistic one. Where a measurement lands inside this [max, sum] interval reads out overlap quality before any profiler is opened, and profilers escalate only on residuals above a stated threshold. Deployment alternatives are compared by wall ordering -- which resource wall binds first as load grows -- rather than by point benchmarks. The account is compositional: new attention or state-space variants enter by declaring one module, and the workflow ships as a zero-dependency calculator plus an agent skill that enforces the discipline in agentic optimization loops. As a case study we analyze a DeepSeek-V3.2-style 671B MoE/MLA model on 16 NVIDIA H20 GPUs, whose ridge point of ~74 FLOP/byte (vs ~590 for H100) makes it an extreme decode-oriented part. The floors show TP16 decoding is KV-capacity-limited to ~70 concurrent 8K requests; sparse attention removes the KV-bandwidth term but not the capacity wall; an EP16+DP-attention layout accepts slightly worse same-batch weight traffic for an order-of-magnitude higher capacity wall (~644) -- while single-stream latency favors TP by 2.4x. The layout judgment is thus a computable function of the operating point, explaining why production deployments on identical hardware have shipped opposite attention layouts.