# Advancing GenAI Assisted Programming--A Comparative Study on Prompt Efficiency and Code Quality Between GPT-4 and GLM-4

arXiv: https://arxiv.org/abs/2402.12782 · HTML: https://mlsanity.com/paper/2402.12782/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 8 of 8)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design](https://mlsanity.com/paper/2303.07839/) (2023)
- [CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models](https://mlsanity.com/paper/2302.00288/) (2023)
- [Better Together? An Evaluation of AI-Supported Code Translation](https://mlsanity.com/paper/2202.07682/) (2022)
- [Human AI Collaboration in Software Engineering: Lessons Learned from a Hands On Workshop](https://mlsanity.com/paper/2312.10620/) (2023)

## Built on by (most cited first, 3 of 3)

- [Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models](https://mlsanity.com/paper/2407.18999/) (2024)
- [Translating Multimodal AI into Real-World Inspection: TEMAI Evaluation Framework and Pathways for Implementation](https://mlsanity.com/paper/2504.13873/) (2025)
- [From Horizontal Layering to Vertical Integration: A Comparative Study of the AI-Driven Software Development Paradigm](https://mlsanity.com/paper/2601.22667/) (2026)

## Abstract (arXiv)

This study aims to explore the best practices for utilizing GenAI as a programming tool, through a comparative analysis between GPT-4 and GLM-4. By evaluating prompting strategies at different levels of complexity, we identify that simplest and straightforward prompting strategy yields best code generation results. Additionally, adding a CoT-like preliminary confirmation step would further increase the success rate. Our results reveal that while GPT-4 marginally outperforms GLM-4, the difference is minimal for average users. In our simplified evaluation model, we see a remarkable 30 to 100-fold increase in code generation efficiency over traditional coding norms. Our GenAI Coding Workshop highlights the effectiveness and accessibility of the prompting methodology developed in this study. We observe that GenAI-assisted coding would trigger a paradigm shift in programming landscape, which necessitates developers to take on new roles revolving around supervising and guiding GenAI, and to focus more on setting high-level objectives and engaging more towards innovation.