# Optimizing AI Inference Across the Deployment Stack

arXiv: https://arxiv.org/abs/2609.10550 · HTML: https://mlsanity.com/paper/2609.10550/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Builds on (most cited first, 13 of 13)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [The Deep Learning Compiler: A Comprehensive Survey](https://mlsanity.com/paper/2002.03794/) (2020)
- ["Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization](https://mlsanity.com/paper/2411.02355/) (2024)
- [AI Benchmark Democratization and Carpentry](https://mlsanity.com/paper/2512.11588/) (2025)
- [KV Cache Optimization Strategies for Scalable and Efficient LLM Inference](https://mlsanity.com/paper/2603.20397/) (2026)

## Built on by (most cited first, 1 of 1)

- [The KV Cache Is the New Memory Wall](https://mlsanity.com/paper/2609.30854/) (2026)

## Abstract (arXiv)

AI deployment performance is shaped not by model architecture alone, but by interactions among compression, compiler transformations, and serving policies. Published benchmarks often report latency and throughput under incomparable conditions, limiting their use for deployment decisions. This paper presents a unified analytical treatment of inference optimization across the deployment stack. We introduce a three-layer taxonomy covering model-level techniques such as quantization, pruning, and distillation; compiler transformations such as graph fusion, layout optimization, and kernel autotuning; and system policies such as dynamic batching, admission control, and memory tiering. We formulate deployment as a constrained multi-objective optimization problem over accuracy, latency, throughput, memory footprint, and energy, and analyze a deployment-ranking functional with Pareto monotonicity and scale invariance. Roofline models show how memory-bandwidth hierarchies bound performance across precision regimes, while queuing models explain how service-time changes amplify response time under load. To improve comparability, we propose an evidence protocol that separates measured, derived, and analytical claims; limits numerical comparison to within-paper results; and requires reporting of hardware, software versions, batch semantics, and thermal state. We synthesize evidence from edge platforms, including Jetson AGX Orin and five inference frameworks; data center GPUs, including A100 and H100 with three LLM serving engines; and quantization studies across the Llama-3.1 family. The synthesis shows that deployment outcomes are governed by cross-layer interactions that no single-layer analysis can predict. We conclude with a constraint-aware selection procedure and open problems in compiler-serving co-optimization, cross-hardware performance prediction, and standardized energy reporting.