mlsanityData to 7 Oct 2026

Synchromesh: Reliable code generation from pre-trained language models

arXiv 2201.11227 ↗First posted 26 Jan 2022cs.LG cs.PLcited by 156

Not on the map. Why?

Abstract (arXiv)

Large pre-trained language models have been used to generate code,providing a flexible interface for synthesizing programs from natural language specifications. However, they often violate syntactic and semantic rules of their output language, limiting their practical usability. In this paper, we propose Synchromesh: a framework for substantially improving the reliability of pre-trained models for code generation. Synchromesh comprises two components. First, it retrieves few-shot examples from a training bank using Target Similarity Tuning (TST), a novel method for semantic example selection. TST learns to recognize utterances that describe similar target programs despite differences in surface natural language features. Then, Synchromesh feeds the examples to a pre-trained language model and samples programs using Constrained Semantic Decoding (CSD): a general framework for constraining the output to a set of valid programs in the target language. CSD leverages constraints on partial outputs to sample complete correct programs, and needs neither re-training nor fine-tuning of the language model. We evaluate our methods by synthesizing code from natural language descriptions using GPT-3 and Codex in three real-world languages: SQL queries, Vega-Lite visualizations and SMCalFlow programs. These domains showcase rich constraints that CSD is able to enforce, including syntax, scope, typing rules, and contextual logic. We observe substantial complementary gains from CSD and TST in prediction accuracy and in effectively preventing run-time errors.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 26 Jan 2022.

Builds on

20 of 30 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  12. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 420
8 more
  1. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  2. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
  3. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  4. 2020SmBoP: Semi-autoregressive Bottom-up Semantic Parsingcited by 97
  5. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  6. 2021In-IDE Code Generation from Natural Language: Promise and Challengescited by 41
  7. 2021Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizationscited by 31
  8. 2016Latent Attention For If-Then Program Synthesiscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 47; 2024: 28; 2025: 39; 2026: 21.

156 papers.

Sort:

By area:

  1. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  2. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  3. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  4. 2023Large Language Model Alignment: A Surveycited by 241
  5. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  6. 2023Efficient Guided Generation for Large Language Modelscited by 192
  7. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  8. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  9. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  10. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  11. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  12. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
38 more
  1. 2023Learning Performance-Improving Code Editscited by 103
  2. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  3. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 99
  4. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 79
  5. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  6. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 73
  7. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  8. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  9. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 66
  10. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 65
  11. 2023Strategic Reasoning with Language Modelscited by 64
  12. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  13. 2024Generating CAD Code with Vision-Language Models for 3D Designscited by 61
  14. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  15. 2022TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Basescited by 59
  16. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 59
  17. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  18. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  19. 2023How to Prompt LLMs for Text-to-SQL: A Study in Zero-shot, Single-domain, and Cross-domain Settingscited by 57
  20. 2023Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repaircited by 55
  21. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  22. 2024Automata-based constraints for language model decodingcited by 49
  23. 2022In-Context Learning for Few-Shot Dialogue State Trackingcited by 47
  24. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  25. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 46
  26. 2024Learning Formal Mathematics From Intrinsic Motivationcited by 45
  27. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  28. 2025Flexible and Efficient Grammar-Constrained Decodingcited by 42
  29. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  30. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  31. 2024PURPLE: Making a Large Language Model a Better SQL Writercited by 34
  32. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 32
  33. 2023Data Formulator: AI-powered Concept-driven Visualization Authoringcited by 31
  34. 2024On the Challenges and Opportunities in Generative AIcited by 31
  35. 2022Repairing Bugs in Python Assignments Using Large Language Modelscited by 28
  36. 2024IterGen: Iterative Semantic-aware Structured LLM Generation with Backtrackingcited by 28
  37. 2024From Language Models over Tokens to Language Models over Characterscited by 28
  38. 2025Learning to Generate Structured Output with Schema Reinforcement Learningcited by 27

50 most cited of 156 All 156 (JSON)