mlsanityData to 7 Oct 2026

Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers

arXiv 2411.09224 ↗First posted 14 Nov 2024cs.SE cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Our everyday lives now heavily rely on artificial intelligence (AI) powered large language models (LLMs). Like regular users, programmers are also benefiting from the newest large language models. In response to the critical role that AI models play in modern software development, this study presents a thorough evaluation of leading programming assistants, including ChatGPT, Gemini(Bard AI), AlphaCode, and GitHub Copilot. The evaluation is based on tasks like natural language processing and code generation accuracy in different programming languages like Java, Python and C++. Based on the results, it has emphasized their strengths and weaknesses and the importance of further modifications to increase the reliability and accuracy of the latest popular models. Although these AI assistants illustrate a high level of progress in language understanding and code generation, along with ethical considerations and responsible usage, they provoke a necessity for discussion. With time, developing more refined AI technology is essential for achieving advanced solutions in various fields, especially with the knowledge of the feature intricacies of these models and their implications. This study offers a comparison of different LLMs and provides essential feedback on the rapidly changing area of AI models. It also emphasizes the need for ethical developmental practices to actualize AI models' full potential.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 15 of its 59 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

15 of 59 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  3. 2023Copyright Violations and Large Language Modelscited by 108
  4. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 100
  5. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  6. 2023Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issuescited by 78
  7. 2023Bias of AI-Generated Content: An Examination of News Produced by Large Language Modelscited by 38
  8. 2023An In-depth Look at Gemini's Language Abilitiescited by 28
  9. 2023Fairness-Aware Structured Pruning in Transformerscited by 21
  10. 2024NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Promptscited by 19
  11. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 19
  12. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 15
3 more
  1. 2024The Pursuit of Fairness in Artificial Intelligence Models: A Surveycited by 13
  2. 2023Introduction to Transformers: an NLP Perspectivecited by 10
  3. 2024GitHub Copilot: the perfect Code compLeeter?cited by 4

Built on by

Cited by year

Citing papers per year: 2025: 5.

5 papers.

  1. 2025Software Reuse in the Generative AI Era: From Cargo Cult Towards AI Native Software Engineeringcited by 2
  2. 2025Behavioral Augmentation of UML Class Diagrams: An Empirical Study of Large Language Models for Method Generationcited by 1
  3. 2025ChatGPT vs. DeepSeek: A Comparative Study on AI-Based Code Generationcited by 0
  4. 2025A Real-Time, Self-Tuning Moderator Framework for Adversarial Prompt Detectioncited by 0
  5. 2025Large Language Models for Education and Research: An Empirical and User Survey-based Analysiscited by 0