# LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages

arXiv: https://arxiv.org/abs/2503.19217 · HTML: https://mlsanity.com/paper/2503.19217/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 24 of 24)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)
- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT](https://mlsanity.com/paper/2304.10778/) (2023)
- [MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages](https://mlsanity.com/paper/2203.08388/) (2022)
- [Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking](https://mlsanity.com/paper/2407.02395/) (2024)
- [Unraveling the Potential of Large Language Models in Code Translation: How Far Are We?](https://mlsanity.com/paper/2410.09812/) (2024)
- [C2HLSC: Leveraging Large Language Models to Bridge the Software-to-Hardware Design Gap](https://mlsanity.com/paper/2412.00214/) (2024)
- [JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Models](https://mlsanity.com/paper/2406.12902/) (2024)
- [ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Models](https://mlsanity.com/paper/2412.17264/) (2024)
- [LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Tasks](https://mlsanity.com/paper/2504.03665/) (2025)
- [Evaluating AI-generated code for C++, Fortran, Go, Java, Julia, Matlab, Python, R, and Rust](https://mlsanity.com/paper/2405.13101/) (2024)
- [Benchmarking the Parallel 1D Heat Equation Solver in Chapel, Charm++, C++, HPX, Go, Julia, Python, Rust, Swift, and Java](https://mlsanity.com/paper/2307.01117/) (2023)
- [Towards a Scalable and Distributed Infrastructure for Deep Learning Applications](https://mlsanity.com/paper/2010.03012/) (2020)

## Built on by (most cited first, 5 of 5)

- [From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflow](https://mlsanity.com/paper/2509.12443/) (2025)
- [Can LLMs Find Bugs in Code? An Evaluation from Beginner Errors to Security Vulnerabilities in Python and C++](https://mlsanity.com/paper/2508.16419/) (2025)
- [How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requests](https://mlsanity.com/paper/2512.24630/) (2025)
- [GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory](https://mlsanity.com/paper/2606.03144/) (2026)
- [From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generation](https://mlsanity.com/paper/2602.22240/) (2026)

## Abstract (arXiv)

The rapid evolution of large language models (LLMs) has opened new possibilities for automating various tasks in software development. This paper evaluates the capabilities of the Llama 2-70B model in automating these tasks for scientific applications written in commonly used programming languages. Using representative test problems, we assess the model's capacity to generate code, documentation, and unit tests, as well as its ability to translate existing code between commonly used programming languages. Our comprehensive analysis evaluates the compilation, runtime behavior, and correctness of the generated and translated code. Additionally, we assess the quality of automatically generated code, documentation and unit tests. Our results indicate that while Llama 2-70B frequently generates syntactically correct and functional code for simpler numerical tasks, it encounters substantial difficulties with more complex, parallelized, or distributed computations, requiring considerable manual corrections. We identify key limitations and suggest areas for future improvements to better leverage AI-driven automation in scientific computing workflows.