mlsanityData to 14 Oct 2026

Formal Algorithms for Model Efficiency

arXiv 2508.14000 ↗First posted 19 Aug 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce the Knob-Meter-Rule (KMR) framework, a unified formalism for representing and reasoning about model efficiency techniques in deep learning. By abstracting diverse methods, including pruning, quantization, knowledge distillation, and parameter-efficient architectures, into a consistent set of controllable knobs, deterministic rules, and measurable meters, KMR provides a mathematically precise and modular perspective on efficiency optimization. The framework enables systematic composition of multiple techniques, flexible policy-driven application, and iterative budgeted optimization through the Budgeted-KMR algorithm. We demonstrate how well-known efficiency methods can be instantiated as KMR triples and present concise algorithmic templates for each. The framework highlights underlying relationships between methods, facilitates hybrid pipelines, and lays the foundation for future research in automated policy learning, dynamic adaptation, and theoretical analysis of cost-quality trade-offs. Overall, KMR offers both a conceptual and practical tool for unifying and advancing model efficiency research.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 23 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

19 of 23 references.

References · 19

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  7. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  8. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  9. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  10. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  11. 2019Importance Estimation for Neural Network Pruningcited by 541
  12. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
7 more
  1. 2015Deep Learning with Limited Numerical Precisioncited by 376
  2. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  3. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  4. 2019Depth-Adaptive Transformercited by 236
  5. 2013Predicting Parameters in Deep Learningcited by 224
  6. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  7. 2023NewsRecLib: A PyTorch-Lightning Library for Neural News Recommendationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Constraint-Driven Model Optimization: An Industry Framework for Selecting Compression and Acceleration Techniques in Modern Machine Learning Systems
  2. 2025Principled Approximation Methods for Efficient and Scalable Deep Learning
  3. 2026Sample-Efficiency of Kolmogorov-Arnold Networks
  4. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning
  5. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks
  6. 2023Knowledge-enhanced Neural Machine Reasoning: A Review
  7. 2025Energy Efficiency in AI for 5G and Beyond: A DeepRx Case Study
  8. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening