# Insights into resource utilization of code small language models serving with runtime engines and execution providers

arXiv: https://arxiv.org/abs/2412.15441 · HTML: https://mlsanity.com/paper/2412.15441/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 86 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model](https://mlsanity.com/paper/2401.02330/) (2024)
- [A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness](https://mlsanity.com/paper/2411.03350/) (2024)
- [Measuring the Carbon Intensity of AI in Cloud Instances](https://mlsanity.com/paper/2206.05229/) (2022)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Impact of Code Language Models on Automated Program Repair](https://mlsanity.com/paper/2302.05020/) (2023)
- [A Systematic Review of Green AI](https://mlsanity.com/paper/2301.11047/) (2023)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [On the Effectiveness of Large Language Models in Domain-Specific Code Generation](https://mlsanity.com/paper/2312.01639/) (2023)
- [Fast and Memory-Efficient Neural Code Completion](https://mlsanity.com/paper/2004.13651/) (2020)
- [A Survey of Small Language Models](https://mlsanity.com/paper/2410.20011/) (2024)
- [ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review](https://mlsanity.com/paper/2305.03123/) (2023)
- [Compressing Pre-trained Models of Code into 3 MB](https://mlsanity.com/paper/2208.07120/) (2022)
- [Greening Large Language Models of Code](https://mlsanity.com/paper/2309.04076/) (2023)

## Abstract (arXiv)

The rapid growth of language models, particularly in code generation, requires substantial computational resources, raising concerns about energy consumption and environmental impact. Optimizing language models inference resource utilization is crucial, and Small Language Models (SLMs) offer a promising solution to reduce resource demands. Our goal is to analyze the impact of deep learning serving configurations, defined as combinations of runtime engines and execution providers, on resource utilization, in terms of energy consumption, execution time, and computing-resource utilization from the point of view of software engineers conducting inference in the context of code generation SLMs. We conducted a technology-oriented, multi-stage experimental pipeline using twelve code generation SLMs to investigate energy consumption, execution time, and computing-resource utilization across the configurations. Significant differences emerged across configurations. CUDA execution provider configurations outperformed CPU execution provider configurations in both energy consumption and execution time. Among the configurations, TORCH paired with CUDA demonstrated the greatest energy efficiency, achieving energy savings from 37.99% up to 89.16% compared to other serving configurations. Similarly, optimized runtime engines like ONNX with the CPU execution provider achieved from 8.98% up to 72.04% energy savings within CPU-based configurations. Also, TORCH paired with CUDA exhibited efficient computing-resource utilization. Serving configuration choice significantly impacts resource utilization. While further research is needed, we recommend the above configurations best suited to software engineers' requirements for enhancing serving resource utilization efficiency.