mlsanityData to 7 Oct 2026

Interactive Code Generation via Test-Driven User-Intent Formalization

arXiv 2208.05950 ↗First posted 11 Aug 2022cs.SE cs.LG cs.PLcited by 50

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown great potential in automating significant aspects of coding by producing natural code from informal natural language (NL) intent. However, when interacting with LLMs, users have no guarantees that the code suggestions produced correctly satisfy the intent they provided. In fact, it is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics. In this paper, we propose the workflow of it interactive test-driven code generation, which leverages lightweight user feedback to (a) formalize the user intent using generated tests that can be useful for debugging, and (b) produce an improved set of code suggestions by pruning and ranking candidate code suggestions. We describe a language-agnostic abstract algorithm and a concrete implementation TiCoder. We perform an automated evaluation of TiCoder on the MBPP and HumanEval code generation benchmarks. Our results are promising with using the OpenAI Codex LLM: our best algorithm improves the passk1 code generation accuracy (in absolute percentages) between 22.49% to 37.71% for MBPP and between 24.79% to 53.98% for HumanEval using between 1 to 5 simulated user queries.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Builds on 18 of its 49 references that are in the map. First posted on arXiv 11 Aug 2022.

Builds on

18 of 49 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  9. 2022CodeT: Code Generation with Generated Testscited by 369
  10. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  11. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  12. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
6 more
  1. 2022Productivity Assessment of Neural Code Completioncited by 97
  2. 2021TOGA: A Neural Method for Test Oracle Generationcited by 57
  3. 2020Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformerscited by 30
  4. 2015A Theory of Formal Synthesis via Inductive Learningcited by 17
  5. 2021Multi-modal Program Inference: a Marriage of Pre-trainedLanguage Models and Component-based Synthesiscited by 12
  6. 2017Interactive Program Synthesiscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 14; 2025: 9; 2026: 14.

50 papers.

Sort:

By area:

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  2. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  3. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  4. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  5. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  6. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  7. 2024CoverUp: Effective High Coverage Test Generation for Pythoncited by 36
  8. 2024Data-driven Discovery with Large Generative Modelscited by 33
  9. 2024Test-Driven Development for Code Generationcited by 32
  10. 2024Automated Proof Generation for Rust Code via Self-Evolutioncited by 31
  11. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 28
  12. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
38 more
  1. 2023Exploring and Characterizing Large Language Models For Embedded System Development and Debuggingcited by 20
  2. 2023Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Modelscited by 19
  3. 2024Towards Neural Synthesis for SMT-Assisted Proof-Oriented Programmingcited by 19
  4. 2022Python Code Generation by Asking Clarification Questionscited by 16
  5. 2024Evaluating LLM-driven User-Intent Formalization for Verification-Aware Languagescited by 16
  6. 2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?cited by 16
  7. 2023Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptionscited by 14
  8. 2025Optimizing Sequential Multi-Step Tasks with Parallel LLM Agentscited by 14
  9. 2023Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Toolscited by 13
  10. 2024B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Testscited by 9
  11. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 9
  12. 2024Data Analysis in the Era of Generative AIcited by 6
  13. 2024Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMscited by 6
  14. 2025Can Code Language Models Learn Clarification-Seeking Behaviors?cited by 5
  15. 2025Beyond Postconditions: Can Large Language Models infer Formal Contracts for Automatic Software Verification?cited by 5
  16. 2023Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesiscited by 4
  17. 2025CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedbackcited by 4
  18. 2024Harnessing the Power of LLMs: Automating Unit Test Generation for High-Performance Computingcited by 3
  19. 2024Examination of Code generated by Large Language Modelscited by 3
  20. 2025Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Inductioncited by 3
  21. 2023Interactive Text Generationcited by 2
  22. 2024Can Large Language Models Code Like a Linguist?: A Case Study in Low Resource Sound Law Inductioncited by 2
  23. 2026A Theoretical Analysis of Test-Driven Code Generationcited by 2
  24. 2026SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inferencecited by 2
  25. 2026Evaluating LLM-Based Test Generation Under Software Evolutioncited by 2
  26. 2025Ambiguity Resolution with Human Feedback for Code Writing Taskscited by 1
  27. 2025Verification Limits Code LLM Trainingcited by 1
  28. 2026Teaching Code LLMs to Reason with Intermediate Formal Specificationscited by 1
  29. 2025Unify and Triumph: Polyglot, Diverse, and Self-Consistent Generation of Unit Tests with LLMscited by 0
  30. 2026MIST-RL: Mutation-based Incremental Suite Testing via Reinforcement Learningcited by 0
  31. 2026Fuzzing with Agents? Generators Are All You Needcited by 0
  32. 2026Interactive Clarification for Cloud Infrastructure-as-Code Synthesiscited by 0
  33. 2026DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocodecited by 0
  34. 2026Specification Grounding Drives Test Effectiveness for LLM Codecited by 0
  35. 2026AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generationcited by 0
  36. 2026VibeJam: A User Study Platform for Web Development with Agentscited by 0
  37. 2026Assurance Envelopes for Autonomous Coding Agents: Minimum-Cost Evidence for Software Changecited by 0
  38. 2026XYEval: Agents say yes to bad advicecited by 0