# Large Language Models for Code Generation: The Practitioners Perspective

arXiv: https://arxiv.org/abs/2501.16998 · HTML: https://mlsanity.com/paper/2501.16998/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2025.

## Built on by (most cited first, 9 of 9)

- [A Cognitive Writing Perspective for Constrained Long-Form Text Generation](https://mlsanity.com/paper/2502.12568/) (2025)
- [TusoAI: Agentic Optimization for Scientific Methods](https://mlsanity.com/paper/2509.23986/) (2025)
- [Wisdom and Delusion of LLM Ensembles for Code Generation and Repair](https://mlsanity.com/paper/2510.21513/) (2025)
- [Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research](https://mlsanity.com/paper/2506.12312/) (2025)
- [MateInfoUB: A Real-World Benchmark for Testing LLMs in Competitive, Multilingual, and Multimodal Educational Tasks](https://mlsanity.com/paper/2507.03162/) (2025)
- [Think Like Human Developers: Harnessing Community Knowledge for Structured Code Reasoning](https://mlsanity.com/paper/2503.14838/) (2025)
- [A Pilot Study on LLM-Based Agentic Translation from Android to iOS: Pitfalls and Insights](https://mlsanity.com/paper/2507.16037/) (2025)
- [Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation](https://mlsanity.com/paper/2608.11513/) (2026)
- [SkillForge: Compositional Skill Synthesis with Verification-in-the-Loop for Generating Formally Verified Dafny Programs](https://mlsanity.com/paper/2608.29841/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have emerged as coding assistants, capable of generating source code from natural language prompts. With the increasing adoption of LLMs in software development, academic research and industry based projects are developing various tools, benchmarks, and metrics to evaluate the effectiveness of LLM-generated code. However, there is a lack of solutions evaluated through empirically grounded methods that incorporate practitioners perspectives to assess functionality, syntax, and accuracy in real world applications. To address this gap, we propose and develop a multi-model unified platform to generate and execute code based on natural language prompts. We conducted a survey with 60 software practitioners from 11 countries across four continents working in diverse professional roles and domains to evaluate the usability, performance, strengths, and limitations of each model. The results present practitioners feedback and insights into the use of LLMs in software development, including their strengths and weaknesses, key aspects overlooked by benchmarks and metrics, and a broader understanding of their practical applicability. These findings can help researchers and practitioners make informed decisions for systematically selecting and using LLMs in software development projects. Future research will focus on integrating more diverse models into the proposed system, incorporating additional case studies, and conducting developer interviews for deeper empirical insights into LLM-driven software development.