# End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference

arXiv: https://arxiv.org/abs/2606.27743 · HTML: https://mlsanity.com/paper/2606.27743/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 26 Jun 2026.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [SliceGPT: Compress Large Language Models by Deleting Rows and Columns](https://mlsanity.com/paper/2401.15024/) (2024)
- [ShortGPT: Layers in Large Language Models are More Redundant Than You Expect](https://mlsanity.com/paper/2403.03853/) (2024)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Mixture-of-Depths: Dynamically allocating compute in transformer-based language models](https://mlsanity.com/paper/2404.02258/) (2024)
- [FastBERT: a Self-distilling BERT with Adaptive Inference Time](https://mlsanity.com/paper/2004.02178/) (2020)
- [LaCo: Large Language Model Pruning via Layer Collapse](https://mlsanity.com/paper/2402.11187/) (2024)
- [Not All Layers of LLMs Are Necessary During Inference](https://mlsanity.com/paper/2403.02181/) (2024)
- [Bridging Discrete and Backpropagation: Straight-Through and Beyond](https://mlsanity.com/paper/2304.08612/) (2023)
- [Fast yet Safe: Early-Exiting with Risk Control](https://mlsanity.com/paper/2405.20915/) (2024)
- [FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models](https://mlsanity.com/paper/2405.18218/) (2024)
- [Adaptive Layer-skipping in Pre-trained LLMs](https://mlsanity.com/paper/2503.23798/) (2025)
- [Learning to Skip for Language Modeling](https://mlsanity.com/paper/2311.15436/) (2023)
- [AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference](https://mlsanity.com/paper/2501.02336/) (2025)
- [SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decoupling](https://mlsanity.com/paper/2506.04179/) (2025)
- [Cascaded Language Models for Cost-effective Human-AI Decision-Making](https://mlsanity.com/paper/2506.11887/) (2025)
- [GR2: Generative Reasoning Re-ranker](https://mlsanity.com/paper/2602.07774/) (2026)
- [DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs](https://mlsanity.com/paper/2407.11030/) (2024)
- [Router-Tuning: A Simple and Effective Approach for Enabling Dynamic-Depth in Transformers](https://mlsanity.com/paper/2410.13184/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs) inference is typically deployed under a static resource assumption, where models execute a fixed computational graph regardless of the runtime environment. However, real-world cloud infrastructure is inherently dynamic, characterized by fluctuating availability (e.g., spot instance preemption) and tiered Quality-of-Service requirements. In such volatile settings, static models are inflexible: they either crash under resource constraints or waste compute on redundant operations. To bridge this gap, we propose Learning to Allocate (L2A), an end-to-end framework for resource-adaptive inference. Unlike prior methods that condition only on input difficulty, we formulate inference as a constrained allocation problem conditioned on both the input and the runtime resource budget itself. We introduce lightweight, budget-conditioned and input-aware gating networks integrated into the LLM. These gates are trained via a unified objective that jointly optimizes task performance, logical consistency, and resource costs along three axes matching how real-world dynamics manifest: layer skipping for memory and depth pressure, head pruning for throughput contention, and reasoning-token reduction for latency tightening. This lets the model learn a budget-aware policy beyond input difficulty alone: it adaptively configures its computational footprint with respect to real-time resource dynamics, maximizing reasoning depth when resources permit while enforcing strict frugality when budgets tighten. A single L2A model traces the entire compute-accuracy Pareto frontier on Llama-3-8B and Qwen-3-4B: at up to 34% realized layer sparsity, it stays within 0.6% of the dense baseline on GSM8K, with the same gap holding zero-shot on out-of-distribution tasks, while every static or heuristic baseline requires a separately tuned model and still drops by 5-10% at comparable inference time.