mlsanityData to 7 Oct 2026

Natural Language Outlines for Code: Literate Programming in the LLM Era

arXiv 2408.04820 ↗First posted 9 Aug 2024cs.SE cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We propose using natural language outlines as a novel modality and interaction surface for providing AI assistance to developers throughout the software development process. An NL outline for a code function comprises multiple statements written in concise prose, which partition the code and summarize its main ideas in the style of literate programming. Crucially, we find that modern LLMs can generate accurate and high-quality NL outlines in practice. Moreover, NL outlines enable a bidirectional sync between code and NL, where a developer can change either code or NL and have the LLM automatically update the other. We discuss many use cases for NL outlines: they can accelerate understanding and navigation of code and diffs, simplify code maintenance, augment code search, steer code generation, and more. We then propose and compare multiple LLM prompting techniques for generating outlines and ask professional developers to judge outline quality. Finally, we present two case studies applying NL outlines toward code review and malware detection.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 44 of its 85 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

44 of 85 references.

References · 44

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  10. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  11. 2022UL2: Unifying Language Learning Paradigmscited by 292
  12. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
32 more
  1. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  2. 2023Using an LLM to Help With Code Understandingcited by 222
  3. 2023Efficient Guided Generation for Large Language Modelscited by 192
  4. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 178
  5. 2020A Transformer-based Approach for Source Code Summarizationcited by 129
  6. 2022Few-shot training LLMs for project-specific code-summarizationcited by 128
  7. 2020Enabling Language Models to Fill in the Blankscited by 121
  8. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  9. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 95
  10. 2022Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Bookcited by 84
  11. 2023Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)cited by 72
  12. 2023Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learningcited by 71
  13. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  14. 2024Automata-based constraints for language model decodingcited by 49
  15. 2023"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Modelscited by 48
  16. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  17. 2020Learning to Update Natural Language Comments Based on Code Changescited by 30
  18. 2023Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Promptscited by 30
  19. 2024Source Code Summarization in the Era of Large Language Modelscited by 30
  20. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  21. 2023Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chainscited by 24
  22. 2023A Comparative Analysis of Large Language Models for Code Documentation Generationcited by 24
  23. 2024Unsupervised Evaluation of Code LLMs with Round-Trip Correctnesscited by 23
  24. 2019Automatic Code Summarization: A Systematic Literature Reviewcited by 18
  25. 2023Developer-Intent Driven Code Comment Generationcited by 16
  26. 2023Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Modelscited by 11
  27. 2024Enhancing LLM-Based Coding Tools through Native Integration of IDE-Derived Static Contextcited by 10
  28. 2014Autofolding for Source Code Summarizationcited by 6
  29. 2024In-IDE Human-AI Experience in the Era of Large Language Models; A Literature Reviewcited by 6
  30. 2024Together We Go Further: LLMs and IDE Static Analysis for Extract Method Refactoringcited by 5
  31. 2023A Study of Documentation for Software Architecturecited by 1
  32. 2023Assessing the Impact of File Ordering Strategies on Code Review Processcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 5.

9 papers.

  1. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  2. 2025ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Groundingcited by 4
  3. 2024MOSS: Enabling Code-Driven Evolution and Context Management for AI Agentscited by 2
  4. 2026AICD Bench: A Challenging Benchmark for AI-Generated Code Detectioncited by 2
  5. 2025NLD-LLM: A systematic framework for evaluating small language transformer models on natural language descriptioncited by 1
  6. 2026LLM-Based Automated Diagnosis Of Integration Test Failures At Googlecited by 0
  7. 2026A Literate Programming Environment for Human and Machine Agentscited by 0
  8. 2026From Reading Code to Reading Spec: A Verified Layer for LLM-Driven Codebase Maintenancecited by 0
  9. 2026Talking to Itself While Coding: What Makes Comments Help Code Generation?cited by 0