# Optimal Neural Program Synthesis from Multimodal Specifications

arXiv: https://arxiv.org/abs/2010.01678 · HTML: https://mlsanity.com/paper/2010.01678/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2020.

## Built on by (most cited first, 15 of 15)

- [SatLM: Satisfiability-Aided Language Models Using Declarative Prompting](https://mlsanity.com/paper/2305.09656/) (2023)
- [Leveraging Language to Learn Program Abstractions and Search Heuristics](https://mlsanity.com/paper/2106.11053/) (2021)
- [Communicating Natural Programs to Humans and Machines](https://mlsanity.com/paper/2106.07824/) (2021)
- [LLMDFA: Analyzing Dataflow in Code with Large Language Models](https://mlsanity.com/paper/2402.10754/) (2024)
- [VerMCTS: Synthesizing Multi-Step Programs using a Verifier, a Large Language Model, and Tree Search](https://mlsanity.com/paper/2402.08147/) (2024)
- [InfeRE: Step-by-Step Regex Generation via Chain of Inference](https://mlsanity.com/paper/2308.04041/) (2023)
- [ImageEye: Batch Image Processing Using Program Synthesis](https://mlsanity.com/paper/2304.03253/) (2023)
- [A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspective](https://mlsanity.com/paper/2202.04847/) (2022)
- [Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction](https://mlsanity.com/paper/2501.16524/) (2025)
- [Synthesizing Programs with Continuous Optimization](https://mlsanity.com/paper/2211.00828/) (2022)
- [Can Large Language Models Code Like a Linguist?: A Case Study in Low Resource Sound Law Induction](https://mlsanity.com/paper/2406.12725/) (2024)
- [ObSynth: An Interactive Synthesis System for Generating Object Models from Natural Language Specifications](https://mlsanity.com/paper/2210.11468/) (2022)
- [MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models](https://mlsanity.com/paper/2509.22151/) (2025)
- [Learning-Based Automatic Synthesis of Software Code and Configuration](https://mlsanity.com/paper/2305.15642/) (2023)
- [Accessible Smart Contracts Verification: Synthesizing Formal Models with Tamed LLMs](https://mlsanity.com/paper/2501.12972/) (2025)

## Abstract (arXiv)

Multimodal program synthesis, which leverages different types of user input to synthesize a desired program, is an attractive way to scale program synthesis to challenging settings; however, it requires integrating noisy signals from the user, like natural language, with hard constraints on the program's behavior. This paper proposes an optimal neural synthesis approach where the goal is to find a program that satisfies user-provided constraints while also maximizing the program's score with respect to a neural model. Specifically, we focus on multimodal synthesis tasks in which the user intent is expressed using a combination of natural language (NL) and input-output examples. At the core of our method is a top-down recurrent neural model that places distributions over abstract syntax trees conditioned on the NL input. This model not only allows for efficient search over the space of syntactically valid programs, but it allows us to leverage automated program analysis techniques for pruning the search space based on infeasibility of partial programs with respect to the user's constraints. The experimental results on a multimodal synthesis dataset (StructuredRegex) show that our method substantially outperforms prior state-of-the-art techniques in terms of accuracy and efficiency, and finds model-optimal programs more frequently.