mlsanityData to 7 Oct 2026

LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages

arXiv 2503.19217 ↗First posted 24 Mar 2025cs.SE cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

The rapid evolution of large language models (LLMs) has opened new possibilities for automating various tasks in software development. This paper evaluates the capabilities of the Llama 2-70B model in automating these tasks for scientific applications written in commonly used programming languages. Using representative test problems, we assess the model's capacity to generate code, documentation, and unit tests, as well as its ability to translate existing code between commonly used programming languages. Our comprehensive analysis evaluates the compilation, runtime behavior, and correctness of the generated and translated code. Additionally, we assess the quality of automatically generated code, documentation and unit tests. Our results indicate that while Llama 2-70B frequently generates syntactically correct and functional code for simpler numerical tasks, it encounters substantial difficulties with more complex, parallelized, or distributed computations, requiring considerable manual corrections. We identify key limitations and suggest areas for future improvements to better leverage AI-driven automation in scientific computing workflows.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

24 of 37 references.

References · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
12 more
  1. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  2. 2023Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPTcited by 71
  3. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  4. 2024Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarkingcited by 26
  5. 2024Unraveling the Potential of Large Language Models in Code Translation: How Far Are We?cited by 17
  6. 2024C2HLSC: Leveraging Large Language Models to Bridge the Software-to-Hardware Design Gapcited by 16
  7. 2024JavaBench: A Benchmark of Object-Oriented Code Generation for Evaluating Large Language Modelscited by 10
  8. 2024ACECode: A Reinforcement Learning Framework for Aligning Code Efficiency and Correctness in Code Language Modelscited by 10
  9. 2025LLM & HPC:Benchmarking DeepSeek's Performance in High-Performance Computing Taskscited by 8
  10. 2024Evaluating AI-generated code for C++, Fortran, Go, Java, Julia, Matlab, Python, R, and Rustcited by 5
  11. 2023Benchmarking the Parallel 1D Heat Equation Solver in Chapel, Charm++, C++, HPX, Go, Julia, Python, Rust, Swift, and Javacited by 3
  12. 2020Towards a Scalable and Distributed Infrastructure for Deep Learning Applicationscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025From Legacy Fortran to Portable Kokkos: An Autonomous Agentic AI Workflowcited by 4
  2. 2025Can LLMs Find Bugs in Code? An Evaluation from Beginner Errors to Security Vulnerabilities in Python and C++cited by 3
  3. 2025How Do Agentic AI Systems Address Performance Optimizations? A BERTopic-Based Analysis of Pull Requestscited by 1
  4. 2026GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theorycited by 1
  5. 2026From Prompts to Performance: Evaluating LLMs for Task-based Parallel Code Generationcited by 0