Compute Where it Counts: Self Optimizing Language Models
On the map: Efficient LLM inference and serving → Early exit and edge inference
Abstract (arXiv)
Efficient LLM inference research has largely focused on reducing the cost of each decoding step (e.g., using quantization, pruning, or sparse attention), typically applying a uniform computation budget to every generated token. In practice, token difficulty varies widely, so static compression can over-compute on easy steps and under-compute on hard ones. We study dynamic budget allocation for autoregressive decoding: learning how much computation to spend per token from within a single model. Self-Optimizing Language Models (SOL) pair a frozen LLM with a lightweight policy network that reads the LLM hidden state and selects a discrete efficiency action at each decode step. Actions can jointly control (i) token-level attention sparsity, (ii) structured activation pruning in the MLP, and (iii) activation quantization bit-width, while leaving the base model weights unchanged. We train the policy with group-relative policy optimization on teacher-forced episodes: the token sequence is fixed, while we sample multiple compute schedules (i.e., "counterfactual" schedules that vary only the efficiency actions for the same token path) and compare their likelihoods under the same supervision. Our reward trades off language-model quality against soft penalties that encourage episode-average budget usage to match a requested target. Across model variants and compute regimes, SOL improves quality at matched budget over static allocation and strong random schedule search, offering a complementary axis for inference-efficiency optimization. SOL discovers a better quality-efficiency pareto-front across all our experiments and improves MMLU accuracy by up to 7.3% over uniform budget allocation strategies.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 42 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
33 of 42 references.
Closest ancestors · 3
Same direction, newest first
- 2024Training-Free Activation Sparsity in Large Language Models26 Aug
- 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding25 Apr
- 2022Confident Adaptive Language Modeling14 Jul
Other references · 30
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2020Big Bird: Transformers for Longer Sequencescited by 1,684
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
- 2020Rethinking Attention with Performerscited by 1,601
- 2019Generating Long Sequences with Sparse Transformerscited by 1,546
- 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
18 more
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
- 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
- 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
- 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
- 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
- 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
- 2019Depth-Adaptive Transformercited by 236
- 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
- 2019Adaptive Attention Span in Transformerscited by 149
- 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 64
- 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 51
- 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
- 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Modelscited by 39
- 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 29
- 2025SplitReason: Learning To Offload Reasoningcited by 11
- 2023FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Searchcited by 6
- 2021All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networkscited by 5
- 2025Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?cited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
- 2026Compressed-Sensing-Guided, Inference-Aware Structured Reduction for Large Language Models
- 2026Learning Adaptive LLM Decoding
- 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
- 2025HAMburger: Accelerating LLM Inference via Token Smashing
- 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction
- 2024LLM Inference Unveiled: Survey and Roofline Model Insights
- 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference