# A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama

arXiv: https://arxiv.org/abs/2405.03616 · HTML: https://mlsanity.com/paper/2405.03616/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 10 of 10)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [How Secure is Code Generated by ChatGPT?](https://mlsanity.com/paper/2304.09655/) (2023)
- [Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues](https://mlsanity.com/paper/2307.12596/) (2023)
- [A Survey on Automated Program Repair Techniques](https://mlsanity.com/paper/2303.18184/) (2023)
- [Measuring the Runtime Performance of C++ Code Written by Humans using GitHub Copilot](https://mlsanity.com/paper/2305.06439/) (2023)
- [Energy-Efficient Algorithms](https://mlsanity.com/paper/1605.08448/) (2016)

## Built on by (most cited first, 8 of 8)

- [Generating Energy-efficient code with LLMs](https://mlsanity.com/paper/2411.10599/) (2024)
- [Energy-Aware Code Generation with LLMs: Benchmarking Small vs. Large Language Models for Sustainable AI Programming](https://mlsanity.com/paper/2508.08332/) (2025)
- [Can We Make Code Green? Understanding Trade-Offs in LLMs vs. Human Code Optimizations](https://mlsanity.com/paper/2503.20126/) (2025)
- [How Do Companies Manage the Environmental Sustainability of AI? An Interview Study About Green AI Efforts and Regulations](https://mlsanity.com/paper/2505.07317/) (2025)
- [Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report](https://mlsanity.com/paper/2608.11965/) (2026)
- [How Do Agentic AI Systems Deal With Software Energy Concerns? A Pull Request-Based Study](https://mlsanity.com/paper/2512.24636/) (2025)
- [Beyond the Need for Speed: Energy-Aware Code Generation via Simulation-Guided Reinforcement Learning](https://mlsanity.com/paper/2607.04577/) (2026)
- [Improving the Energy-Efficiency of the Code Generated by LLMs through Effective Prompting](https://mlsanity.com/paper/2610.02571/) (2026)

## Abstract (arXiv)

Context. Nowadays, 83% of software developers use Large Language Models (LLMs) to generate code. LLMs recently became essential to increase the productivity of software developers and decrease the time and cost of software development. Developers ranging from novices to experts use LLM tools not only to detect and patch bugs, but also to integrate generated code into their software. However, as of today there is no objective assessment of the energy efficiency of the source code generated by LLM tools. Released in August 2023, Code Llama is one of the most recent LLM tools. Goal. In this paper, we present an empirical study that assesses the energy efficiency of Code Llama with respect to human-written source code. Method. We design an experiment involving three human-written benchmarks implemented in C++, JavaScript, and Python. We ask Code Llama to generate the code of the benchmarks using different prompts and temperatures. Therefore, we execute both implementations and profile their energy efficiency. Results. Our study shows that the energy efficiency of code generated by Code Llama is heavily-dependent on the chosen programming language and the specific code problem at hand. Also, human implementations tend to be more energy efficient overall, with generated JavaScript code outperforming its human counterpart. Moreover, explicitly asking Code Llama to generate energy-efficient code results in an equal or worse energy efficiency, as well as using different temperatures seems not to affect the energy efficiency of generated code. Conclusions. According to our results, code generated using Code Llama does not guarantee energy efficiency, even when prompted to do so. Therefore, software developers should evaluate the energy efficiency of generated code before integrating it into the software system under development.