mlsanityData to 14 Oct 2026

Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decoding

arXiv 2609.04238 ↗First posted 29 Jul 2026cs.ARcited by 0

Not on the map. Why?

Abstract (arXiv)

Autoregressive decoding on cheap hardware is bound not by FLOPs but by the bytes each generated token must move across the slowest populated tier of a memory hierarchy. We treat bytes-per-token as a first-class design axis, organized by an address-determinism taxonomy that classifies parameters by when their fetch address becomes known during a token's forward pass (A0: at token sampling; A1: before attention; A2: layerwise data-dependent; A3: always read). This reduces prefetch scheduling to single-machine feasibility with release times, yielding a closed-form windowed roofline. We run dual-budget (bytes-per-token times storage) ablations across three sub-100M scales, then take the framework to real large-MoE deployment and report a substantial negative result the roofline predicts: on an 8GB edge board running Qwen3-30B-A3B (4-bit, 18GB), the model overflows RAM and decode is pinned at the eMMC bandwidth ceiling; predictive expert prefetch does not help -- not temporal-locality prefetch (net-negative), not even a trace-driven oracle with perfect prediction -- because the binding constraint is byte volume over a saturated bus, which prefetch cannot reduce. The lever that works is reducing bytes-per-token until the model fits the fast tier: quantized to fit a 16GB unified-memory device, the same model runs GPU-resident at 11.5 tok/s (22x). We reconcile this with GPU-serving expert-prefetch predictors: a frozen-model probe predicts Qwen3-30B routing from the pre-attention state at 91.2%, a scale-invariant predictability property, but this converts to throughput only where the fast tier caches most of the model and per-token transfer is comparable to compute -- measured to hold on an A100 PCIe-offload path and to fail on bandwidth-walled edge storage. Predictability is not speedup; we chart where the gap closes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 18 references that are in the map. First posted on arXiv 29 Jul 2026.

Builds on

14 of 18 references.

References · 14

  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  4. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  5. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  7. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  8. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 40
  9. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  10. 2025Mixture of Lookup Expertscited by 12
  11. 2025SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deploymentcited by 5
  12. 2025PT2-LLM: Post-Training Ternarization for Large Language Modelscited by 4
2 more
  1. 2025Mixture of Lookup Key-Value Expertscited by 1
  2. 2026Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Studycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
  2. 2026Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
  3. 2026Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
  4. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding
  5. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration
  6. 2026NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
  7. 2025From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
  8. 2025Architecting Long-Context LLM Acceleration with Packing-Prefetch Scheduler and Ultra-Large Capacity On-Chip Memories