mlsanityData to 7 Oct 2026

TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition

arXiv 2603.12465 ↗First posted 12 Mar 2026cs.DC cs.LG cs.PFcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Large Language Model (LLM) inference is widely used in interactive assistants and agentic systems. In latency-sensitive deployments, inference time can become dominated by host-side overheads. Existing approaches typically expose this cost only as an aggregate residual or a launch/queue metric, which is often insufficient to identify which execution layer should be optimized. This work presents TaxBreak, a trace-driven methodology for decomposing host-visible orchestration overhead into three components: framework translation time, CUDA library translation time, and kernel launch-path time. We validate TaxBreak on NVIDIA H100 and H200 systems and use it to derive our proposed Host-Device Balance Index (HDBI), a boundedness summary index that relates device-active execution to host-visible orchestration. Across representative dense and mixture-of-experts workloads in both prefill and decode, we show that aggregate latency, GPU inactivity, or boundedness ratios alone can obscure the dominant optimization target. TaxBreak instead distinguishes cases where optimization should reduce software-stack overhead from cases where the primary win comes from reducing device-side work. We further show that MoE models dispatch 8-11x more kernels per output token than dense models, and that for such host-bound workloads, CPU single-thread performance is a first-order parameter: a faster host CPU reduces orchestration overhead by 10-29% and improves end-to-end latency by up to 14%, even when paired with a slower-clocked GPU. These results position TaxBreak as a diagnostic tool for assessing whether optimization effort should target the software stack or the device-side workload execution.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 37 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

27 of 37 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
15 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 115
  3. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  4. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Modelscited by 19
  6. 2025FlashMoE: Fast Distributed MoE in a Single Kernelcited by 13
  7. 2023The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deploymentcited by 11
  8. 2025LIMINAL: Exploring The Frontiers of LLM Decode Performancecited by 9
  9. 2025Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architecturescited by 8
  10. 2025A Systematic Characterization of LLM Inference on GPUscited by 8
  11. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Modelscited by 6
  12. 2020AI Tax: The Hidden Cost of AI Data Center Applicationscited by 3
  13. 2022Machine Learning Systems are Bloated and Vulnerablecited by 2
  14. 2025Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMscited by 2
  15. 2025The Hidden Bloat in Machine Learning Systemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Characterization of Request and Token Energy Costs for LLM Inference Workloads on GPU Platformscited by 2
  2. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows
  2. 2026Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
  3. 2026From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving
  4. 2026KernelSight-LM: A Kernel-Level LLM Inference Simulator
  5. 2026LLM4LLM: Bridging Kernel Benchmarks and Real Deployment via Closed-Loop Agentic Optimization
  6. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  7. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  8. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains