# Program Synthesis Dialog Agents for Interactive Decision-Making

arXiv: https://arxiv.org/abs/2502.19610 · HTML: https://mlsanity.com/paper/2502.19610/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 17 of 17)

- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [Task-Oriented Dialogue as Dataflow Synthesis](https://mlsanity.com/paper/2009.11423/) (2020)
- [CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models](https://mlsanity.com/paper/2305.14318/) (2023)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Interpretation of Natural Language Rules in Conversational Machine Reading](https://mlsanity.com/paper/1809.01494/) (2018)
- [User Modeling for Task Oriented Dialogues](https://mlsanity.com/paper/1811.04369/) (2018)
- [DynaSaur: Large Language Agents Beyond Predefined Actions](https://mlsanity.com/paper/2411.01747/) (2024)
- [ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog](https://mlsanity.com/paper/2409.06097/) (2024)
- [Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit](https://mlsanity.com/paper/2401.00288/) (2023)
- [Towards Asking Clarification Questions for Information Seeking on Task-Oriented Dialogues](https://mlsanity.com/paper/2305.13690/) (2023)
- [Symbolic Planning and Code Generation for Grounded Dialogue](https://mlsanity.com/paper/2310.17140/) (2023)
- [Alexpaca: Learning Factual Clarification Question Generation Without Examples](https://mlsanity.com/paper/2310.11571/) (2023)

## Built on by (most cited first, 1 of 1)

- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)

## Abstract (arXiv)

Many real-world eligibility problems, ranging from medical diagnosis to tax planning, can be mapped to decision problems expressed in natural language, wherein a model must make a binary choice based on user features. Large-scale domains such as legal codes or frequently updated funding opportunities render human annotation (e.g., web forms or decision trees) impractical, highlighting the need for agents that can automatically assist in decision-making. Since relevant information is often only known to the user, it is crucial that these agents ask the right questions. As agents determine when to terminate a conversation, they face a trade-off between accuracy and the number of questions asked, a key metric for both user experience and cost. To evaluate this task, we propose BeNYfits, a new benchmark for determining user eligibility for multiple overlapping social benefits opportunities through interactive decision-making. Our experiments show that current language models struggle with frequent hallucinations, with GPT-4o scoring only 35.7 F1 using a ReAct-style chain-of-thought. To address this, we introduce ProADA, a novel approach that leverages program synthesis to assist in decision-making by mapping dialog planning to a code generation problem and using gaps in structured data to determine the best next action. Our agent, ProADA, improves the F1 score to 55.6 while maintaining nearly the same number of dialog turns.