# In-IDE Code Generation from Natural Language: Promise and Challenges

arXiv: https://arxiv.org/abs/2101.11149 · HTML: https://mlsanity.com/paper/2101.11149/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2021.

## Built on by (most cited first, 30 of 42)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Clover: Closed-Loop Verifiable Code Generation](https://mlsanity.com/paper/2310.17807/) (2023)
- [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022)
- [MCU: An Evaluation Framework for Open-Ended Game Agents](https://mlsanity.com/paper/2310.08367/) (2023)
- [Refactoring Programs Using Large Language Models with Few-Shot Examples](https://mlsanity.com/paper/2311.11690/) (2023)
- [Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation](https://mlsanity.com/paper/2305.00909/) (2023)
- [Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking](https://mlsanity.com/paper/2402.08030/) (2024)
- [A Hazard Analysis Framework for Code Synthesis Large Language Models](https://mlsanity.com/paper/2207.14157/) (2022)
- [Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review](https://mlsanity.com/paper/2406.12655/) (2024)
- [ANPL: Towards Natural Programming with Interactive Decomposition](https://mlsanity.com/paper/2305.18498/) (2023)
- [Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks](https://mlsanity.com/paper/2402.05650/) (2024)
- [Toward Code Generation: A Survey and Lessons from Semantic Parsing](https://mlsanity.com/paper/2105.03317/) (2021)
- [Reading StackOverflow Encourages Cheating: Adding Question Text Improves Extractive Code Generation](https://mlsanity.com/paper/2106.04447/) (2021)
- [AutoCoder: Enhancing Code Large Language Model with textscAIEV-Instruct](https://mlsanity.com/paper/2405.14906/) (2024)
- [Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming](https://mlsanity.com/paper/2407.20712/) (2024)
- [Architext: Language-Driven Generative Architecture Design](https://mlsanity.com/paper/2303.07519/) (2023)
- [Towards Advancing Code Generation with Large Language Models: A Research Roadmap](https://mlsanity.com/paper/2501.11354/) (2025)
- [Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software](https://mlsanity.com/paper/2504.15549/) (2025)
- [Automatic Document Sketching: Generating Drafts from Analogous Texts](https://mlsanity.com/paper/2106.07192/) (2021)
- [Natural Language-Guided Programming](https://mlsanity.com/paper/2108.05198/) (2021)
- [Cascaded Fast and Slow Models for Efficient Semantic Code Search](https://mlsanity.com/paper/2110.07811/) (2021)
- [The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study](https://mlsanity.com/paper/2507.03156/) (2025)
- [A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective](https://mlsanity.com/paper/2202.04847/) (2022)
- [NS3: Neuro-Symbolic Semantic Code Search](https://mlsanity.com/paper/2205.10674/) (2022)
- [AI for Low-Code for AI](https://mlsanity.com/paper/2305.20015/) (2023)

## Abstract (arXiv)

A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs. One major difficulty of programming is turning concept into code, especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries, but these have primarily been evaluated purely based on retrieval accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested. We perform the first comprehensive investigation of the promise and challenges of using such technology inside the IDE, asking "at the current state of technology does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?" We first develop a plugin for the IDE that implements a hybrid of code generation and code retrieval functionality, and orchestrate virtual environments to enable collection of many user events. We ask developers with various backgrounds to complete 14 Python programming tasks ranging from basic file manipulation to machine learning or data visualization, with or without the help of the plugin. While qualitative surveys of developer experience are largely positive, quantitative results with regards to increased productivity, code quality, or program correctness are inconclusive. Analysis identifies several pain points that could improve the effectiveness of future machine learning based code generation/retrieval developer assistants, and demonstrates when developers prefer code generation over code retrieval and vice versa. We release all data and software to pave the road for future empirical studies and development of better models.