# Natural Language Outlines for Code: Literate Programming in the LLM Era

arXiv: https://arxiv.org/abs/2408.04820 · HTML: https://mlsanity.com/paper/2408.04820/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 85 references that are in the map.
- First posted on arXiv 9 Aug 2024.

## Builds on (most cited first, 30 of 44)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Efficient Guided Generation for Large Language Models](https://mlsanity.com/paper/2307.09702/) (2023)
- [Large Language Models for Software Engineering: Survey and Open Problems](https://mlsanity.com/paper/2310.03533/) (2023)
- [A Transformer-based Approach for Source Code Summarization](https://mlsanity.com/paper/2005.00653/) (2020)
- [Few-shot training LLMs for project-specific code-summarization](https://mlsanity.com/paper/2207.04237/) (2022)
- [Enabling Language Models to Fill in the Blanks](https://mlsanity.com/paper/2005.05339/) (2020)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)](https://mlsanity.com/paper/2304.06815/) (2023)
- [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [Automata-based constraints for language model decoding](https://mlsanity.com/paper/2407.08103/) (2024)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- [Skill-Based Few-Shot Selection for In-Context Learning](https://mlsanity.com/paper/2305.14210/) (2023)
- [Learning to Update Natural Language Comments Based on Code Changes](https://mlsanity.com/paper/2004.12169/) (2020)
- [Source Code Summarization in the Era of Large Language Models](https://mlsanity.com/paper/2407.07959/) (2024)

## Built on by (most cited first, 9 of 9)

- [Large Language Models (LLMs) for Source Code Analysis: applications, models and datasets](https://mlsanity.com/paper/2503.17502/) (2025)
- [ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding](https://mlsanity.com/paper/2504.00019/) (2025)
- [MOSS: Enabling Code-Driven Evolution and Context Management for AI Agents](https://mlsanity.com/paper/2409.16120/) (2024)
- [AICD Bench: A Challenging Benchmark for AI-Generated Code Detection](https://mlsanity.com/paper/2602.02079/) (2026)
- [NLD-LLM: A systematic framework for evaluating small language transformer models on natural language description](https://mlsanity.com/paper/2510.05139/) (2025)
- [LLM-Based Automated Diagnosis Of Integration Test Failures At Google](https://mlsanity.com/paper/2604.12108/) (2026)
- [A Literate Programming Environment for Human and Machine Agents](https://mlsanity.com/paper/2608.24644/) (2026)
- [From Reading Code to Reading Spec: A Verified Layer for LLM-Driven Codebase Maintenance](https://mlsanity.com/paper/2609.06383/) (2026)
- [Talking to Itself While Coding: What Makes Comments Help Code Generation?](https://mlsanity.com/paper/2609.09242/) (2026)

## Abstract (arXiv)

We propose using natural language outlines as a novel modality and interaction surface for providing AI assistance to developers throughout the software development process. An NL outline for a code function comprises multiple statements written in concise prose, which partition the code and summarize its main ideas in the style of literate programming. Crucially, we find that modern LLMs can generate accurate and high-quality NL outlines in practice. Moreover, NL outlines enable a bidirectional sync between code and NL, where a developer can change either code or NL and have the LLM automatically update the other. We discuss many use cases for NL outlines: they can accelerate understanding and navigation of code and diffs, simplify code maintenance, augment code search, steer code generation, and more. We then propose and compare multiple LLM prompting techniques for generating outlines and ask professional developers to judge outline quality. Finally, we present two case studies applying NL outlines toward code review and malware detection.