# Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy

arXiv: https://arxiv.org/abs/2412.00329 · HTML: https://mlsanity.com/paper/2412.00329/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2024.

## Built on by (most cited first, 3 of 3)

- [Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices](https://mlsanity.com/paper/2506.01774/) (2025)
- [Aggregating empirical evidence from data strategy studies: a case on model quantization](https://mlsanity.com/paper/2505.00816/) (2025)
- [Comparing energy consumption and accuracy in text classification inference](https://mlsanity.com/paper/2508.14170/) (2025)

## Abstract (arXiv)

The use of generative AI-based coding assistants like ChatGPT and Github Copilot is a reality in contemporary software development. Many of these tools are provided as remote APIs. Using third-party APIs raises data privacy and security concerns for client companies, which motivates the use of locally-deployed language models. In this study, we explore the trade-off between model accuracy and energy consumption, aiming to provide valuable insights to help developers make informed decisions when selecting a language model. We investigate the performance of 18 families of LLMs in typical software development tasks on two real-world infrastructures, a commodity GPU and a powerful AI-specific GPU. Given that deploying LLMs locally requires powerful infrastructure which might not be affordable for everyone, we consider both full-precision and quantized models. Our findings reveal that employing a big LLM with a higher energy budget does not always translate to significantly improved accuracy. Additionally, quantized versions of large models generally offer better efficiency and accuracy compared to full-precision versions of medium-sized ones. Apart from that, not a single model is suitable for all types of software development tasks.