# From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper

arXiv: https://arxiv.org/abs/2503.07450 · HTML: https://mlsanity.com/paper/2503.07450/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 79 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Do LLMs exhibit human-like response biases? A case study in survey design](https://mlsanity.com/paper/2311.04076/) (2023)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [Structured Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2305.06599/) (2023)
- [PentestGPT: An LLM-empowered Automatic Penetration Testing Tool](https://mlsanity.com/paper/2308.06782/) (2023)
- [Verbosity Bias in Preference Labeling by Large Language Models](https://mlsanity.com/paper/2310.10076/) (2023)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [Disclosure and Mitigation of Gender Bias in LLMs](https://mlsanity.com/paper/2402.11190/) (2024)
- [Summarization is (Almost) Dead](https://mlsanity.com/paper/2309.09558/) (2023)
- [Program Synthesis using Natural Language](https://mlsanity.com/paper/1509.00413/) (2015)
- [Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study](https://mlsanity.com/paper/2404.13414/) (2024)
- [On the use of BERT for Neural Machine Translation](https://mlsanity.com/paper/1909.12744/) (2019)
- [NER-BERT: A Pre-trained Model for Low-Resource Entity Tagging](https://mlsanity.com/paper/2112.00405/) (2021)
- [Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming](https://mlsanity.com/paper/2306.05153/) (2023)
- [LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities](https://mlsanity.com/paper/2309.13574/) (2023)
- [A deep language model for software code](https://mlsanity.com/paper/1608.02715/) (2016)

## Built on by (most cited first, 3 of 3)

- [LLM-CSEC: Empirical Evaluation of Security in C/C++ Code Generated by Large Language Models](https://mlsanity.com/paper/2511.18966/) (2025)
- [From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Model](https://mlsanity.com/paper/2601.19053/) (2026)
- [A Showdown of ChatGPT vs DeepSeek in Solving Programming Tasks](https://mlsanity.com/paper/2503.13549/) (2025)

## Abstract (arXiv)

The introduction of transformer architecture was a turning point in Natural Language Processing (NLP). Models based on the transformer architecture such as Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-Trained Transformer (GPT) have gained widespread popularity in various applications such as software development and education. The availability of Large Language Models (LLMs) such as ChatGPT and Bard to the general public has showcased the tremendous potential of these models and encouraged their integration into various domains such as software development for tasks such as code generation, debugging, and documentation generation. In this study, opinions from 11 experts regarding their experience with LLMs for software development have been gathered and analysed to draw insights that can guide successful and responsible integration. The overall opinion of the experts is positive, with the experts identifying advantages such as increase in productivity and reduced coding time. Potential concerns and challenges such as risk of over-dependence and ethical considerations have also been highlighted.