# What is it like to program with artificial intelligence? arXiv: https://arxiv.org/abs/2208.06213 ยท HTML: https://mlsanity.com/paper/2208.06213/ ## Key facts - Cited by 39 papers in the map (data through 14 Oct 2026). - Builds on 20 of its 97 references that are in the map. - First posted on arXiv 12 Aug 2022. ## Builds on (most cited first, 20 of 20) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013) - [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014) - [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021) - [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022) - [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017) - [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022) - [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022) - [Retrieval-Augmented Generation for Code Summarization via Hybrid GNN](https://mlsanity.com/paper/2006.05405/) (2020) - [LambdaNet: Probabilistic Type Inference using Graph Neural Networks](https://mlsanity.com/paper/2005.02161/) (2020) - [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021) - [Is explainable AI a race against model complexity?](https://mlsanity.com/paper/2205.10119/) (2022) - [SmartPaste: Learning to Adapt Source Code](https://mlsanity.com/paper/1705.07867/) (2017) ## Built on by (most cited first, 30 of 39) - [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023) - [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022) - [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023) - [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023) - [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://mlsanity.com/paper/2407.07064/) (2024) - [Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition](https://mlsanity.com/paper/2407.02651/) (2024) - [Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking](https://mlsanity.com/paper/2402.08030/) (2024) - [When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming](https://mlsanity.com/paper/2306.04930/) (2023) - [Bringing Generative AI to Adaptive Learning in Education](https://mlsanity.com/paper/2402.14601/) (2024) - [HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent](https://mlsanity.com/paper/2406.00215/) (2024) - [Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions](https://mlsanity.com/paper/2302.07248/) (2023) - [Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming](https://mlsanity.com/paper/2306.05153/) (2023) - [HPC-Coder: Modeling Parallel Programs using Large Language Models](https://mlsanity.com/paper/2306.17281/) (2023) - [Exploring Perspectives on the Impact of Artificial Intelligence on the Creativity of Knowledge Work: Beyond Mechanised Plagiarism and Stochastic Parrots](https://mlsanity.com/paper/2307.10751/) (2023) - [Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2402.11702/) (2024) - [How Do Programming Students Use Generative AI?](https://mlsanity.com/paper/2501.10091/) (2025) - [Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code](https://mlsanity.com/paper/2312.14856/) (2023) - [A Language Model of Java Methods with Train/Test Deduplication](https://mlsanity.com/paper/2305.08286/) (2023) - [Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming](https://mlsanity.com/paper/2407.20712/) (2024) - [Co-audit: tools to help humans double-check AI-generated content](https://mlsanity.com/paper/2310.01297/) (2023) - [Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software](https://mlsanity.com/paper/2504.15549/) (2025) - [Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation](https://mlsanity.com/paper/2504.18804/) (2025) - [NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification](https://mlsanity.com/paper/2508.02823/) (2025) - [Humans are Missing from AI Coding Agent Research](https://mlsanity.com/paper/2608.12355/) (2026) - [Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model](https://mlsanity.com/paper/2608.13867/) (2026) - [Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes](https://mlsanity.com/paper/2409.06416/) (2024) - [Optimizing AI-Assisted Code Generation](https://mlsanity.com/paper/2412.10953/) (2024) - [Evaluating the Process Modeling Abilities of Large Language Models -- Preliminary Foundations and Results](https://mlsanity.com/paper/2503.13520/) (2025) - [AutoCodeSherpa: Symbolic Explanations in AI Coding Agents](https://mlsanity.com/paper/2507.22414/) (2025) - [The Variance Paradox: How AI Reduces Diversity but Increases Novelty](https://mlsanity.com/paper/2508.19264/) (2025) ## Abstract (arXiv) Large language models, such as OpenAI's codex and Deepmind's AlphaCode, can generate code to solve a variety of problems expressed in natural language. This technology has already been commercialised in at least one widely-used programming editor extension: GitHub Copilot. In this paper, we explore how programming with large language models (LLM-assisted programming) is similar to, and differs from, prior conceptualisations of programmer assistance. We draw upon publicly available experience reports of LLM-assisted programming, as well as prior usability and design studies. We find that while LLM-assisted programming shares some properties of compilation, pair programming, and programming via search and reuse, there are fundamental differences both in the technical possibilities as well as the practical experience. Thus, LLM-assisted programming ought to be viewed as a new way of programming with its own distinct properties and challenges. Finally, we draw upon observations from a user study in which non-expert end user programmers use LLM-assisted tools for solving data tasks in spreadsheets. We discuss the issues that might arise, and open research challenges, in applying large language models to end-user programming, particularly with users who have little or no programming expertise.