mlsanityData to 14 Oct 2026

Jigsaw: Large Language Models meet Program Synthesis

arXiv 2112.02969 ↗First posted 6 Dec 2021cs.SE cs.PLcited by 103

Not on the map. Why?

Abstract (arXiv)

Large pre-trained language models such as GPT-3, Codex, and Google's language model are now capable of generating code from natural language specifications of programmer intent. We view these developments with a mixture of optimism and caution. On the optimistic side, such large language models have the potential to improve productivity by providing an automated AI pair programmer for every programmer in the world. On the cautionary side, since these large language models do not understand program semantics, they offer no guarantees about quality of the suggested code. In this paper, we present an approach to augment these large language models with post-processing steps based on program analysis and synthesis techniques, that understand the syntax and semantics of programs. Further, we show that such techniques can make use of user feedback and improve with usage. We present our experiences from building and evaluating such a tool jigsaw, targeted at synthesizing code for using Python Pandas API using multi-modal inputs. Our experience suggests that as these large language models evolve for synthesizing code from intent, jigsaw has an important role to play in improving the accuracy of the systems.

Key facts. Cited by 103 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 36; 2024: 30; 2025: 20; 2026: 6.

103 papers.

Sort:

By area:

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  3. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  4. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  5. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  6. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 148
  7. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  8. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
  9. 2023Learning Performance-Improving Code Editscited by 106
  10. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 102
  11. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  12. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 86
38 more
  1. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 76
  2. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 64
  3. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  4. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  5. 2022When Language Model Meets Private Librarycited by 54
  6. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  7. 2022Interactive Code Generation via Test-Driven User-Intent Formalizationcited by 50
  8. 2023Large Language Models and Simple, Stupid Bugscited by 45
  9. 2023A New Era in Software Security: Towards Self-Healing Software via Large Language Models and Formal Verificationcited by 42
  10. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  11. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  12. 2023Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Modelscited by 38
  13. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 34
  14. 2024CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?cited by 30
  15. 2023Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Modelscited by 28
  16. 2024Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4cited by 28
  17. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  18. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  19. 2024MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generationcited by 26
  20. 2023AceCoder: Utilizing Existing Code to Enhance Code Generationcited by 25
  21. 2024Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Taskscited by 25
  22. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 25
  23. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveragingcited by 25
  24. 2023BOLT: Fast Energy-based Controlled Text Generation with Tunable Biasescited by 24
  25. 2024How secure is AI-generated Code: A Large-Scale Comparison of Large Language Modelscited by 24
  26. 2022Towards a Mathematics Formalisation Assistant using Large Language Modelscited by 22
  27. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
  28. 2023Exploring and Characterizing Large Language Models For Embedded System Development and Debuggingcited by 20
  29. 2024Turn-taking and Backchannel Prediction with Acoustic and Large Language Model Fusioncited by 20
  30. 2024X-lifecycle Learning for Cloud Incident Management using LLMscited by 19
  31. 2023Augmenting Greybox Fuzzing with Generative AIcited by 18
  32. 2024SymbolicAI: A framework for logic-based approaches combining generative models and solverscited by 16
  33. 2024Syzygy: Dual Code-Test C to (safe) Rust Translation using LLMs and Dynamic Analysiscited by 16
  34. 2023InstructExcel: A Benchmark for Natural Language Instruction in Excelcited by 15
  35. 2022Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectorscited by 14
  36. 2023From Copilot to Pilot: Towards AI Supported Software Developmentcited by 14
  37. 2023From Words to Code: Harnessing Data for Program Synthesis from Natural Languagecited by 11
  38. 2024A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentationcited by 10

50 most cited of 103 All 103 (JSON)