# From Copilot to Pilot: Towards AI Supported Software Development

arXiv: https://arxiv.org/abs/2303.04142 · HTML: https://mlsanity.com/paper/2303.04142/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 14 of 14)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Jigsaw: Large Language Models meet Program Synthesis](https://mlsanity.com/paper/2112.02969/) (2021)
- [What do pre-trained code models know about code?](https://mlsanity.com/paper/2108.11308/) (2021)
- [CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning](https://mlsanity.com/paper/1904.00720/) (2019)
- [An Empirical Study on the Usage of BERT Models for Code Completion](https://mlsanity.com/paper/2103.07115/) (2021)
- [To What Extent do Deep Learning-based Code Recommenders Generate Predictions by Cloning Code from the Training Set?](https://mlsanity.com/paper/2204.06894/) (2022)
- [SOTorrent: Reconstructing and Analyzing the Evolution of Stack Overflow Posts](https://mlsanity.com/paper/1803.07311/) (2018)
- [What to Fix? Distinguishing between design and non-design rules in automated tools](https://mlsanity.com/paper/1705.11087/) (2017)
- [A Systematic Review of Productivity Factors in Software Development](https://mlsanity.com/paper/1801.06475/) (2018)
- [A Program Synthesis Approach for Adding Architectural Tactics to An Existing Code Base](https://mlsanity.com/paper/2108.07117/) (2021)

## Built on by (most cited first, 14 of 14)

- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- ["I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust](https://mlsanity.com/paper/2405.00623/) (2024)
- [Generative Artificial Intelligence: A Systematic Review and Applications](https://mlsanity.com/paper/2405.11029/) (2024)
- [How Developers Interact with AI: A Taxonomy of Human-AI Collaboration in Software Engineering](https://mlsanity.com/paper/2501.08774/) (2025)
- [From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Education](https://mlsanity.com/paper/2509.21972/) (2025)
- [Natural Language-Oriented Programming (NLOP): Towards Democratizing Software Creation](https://mlsanity.com/paper/2406.05409/) (2024)
- [The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development](https://mlsanity.com/paper/2605.01160/) (2026)
- [LLM-Assisted Repository-Level Generation with Structured Spec-Driven Engineering](https://mlsanity.com/paper/2605.02455/) (2026)
- [On Languaging a Simulation Engine](https://mlsanity.com/paper/2402.16482/) (2024)
- [PeerCoPilot: A Language Model-Powered Assistant for Behavioral Health Organizations](https://mlsanity.com/paper/2511.21721/) (2025)
- [Learning to Code with Context: A Study-Based Approach](https://mlsanity.com/paper/2512.05242/) (2025)
- [Artificial Intelligence as a Catalyst for Innovation in Software Engineering](https://mlsanity.com/paper/2603.10994/) (2026)
- [Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality](https://mlsanity.com/paper/2607.14816/) (2026)
- [From Overload to Insights: How AI Agents Can Support Scientists in Analyzing Complex Data](https://mlsanity.com/paper/2607.16845/) (2026)

## Abstract (arXiv)

AI-supported programming has arrived, as shown by the introduction and successes of large language models for code, such as Copilot/Codex (Github/OpenAI) and AlphaCode (DeepMind). Above human average performance on programming challenges is now possible. However, software engineering is much more than solving programming contests. Moving beyond code completion to AI-supported software engineering will require an AI system that can, among other things, understand how to avoid code smells, to follow language idioms, and eventually (maybe!) propose rational software designs. In this study, we explore the current limitations of AI-supported code completion tools like Copilot and offer a simple taxonomy for understanding the classification of AI-supported code completion tools in this space. We first perform an exploratory study on Copilot's code suggestions for language idioms and code smells. Copilot does not follow language idioms and avoid code smells in most of our test scenarios. We then conduct additional investigation to determine the current boundaries of AI-supported code completion tools like Copilot by introducing a taxonomy of software abstraction hierarchies where 'basic programming functionality' such as code compilation and syntax checking is at the least abstract level, software architecture analysis and design are at the most abstract level. We conclude by providing a discussion on challenges for future development of AI-supported code completion tools to reach the design level of abstraction in our taxonomy.