# Symbolic Planning and Code Generation for Grounded Dialogue

arXiv: https://arxiv.org/abs/2310.17140 · HTML: https://mlsanity.com/paper/2310.17140/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Decoupling Strategy and Generation in Negotiation Dialogues](https://mlsanity.com/paper/1808.09637/) (2018)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)](https://mlsanity.com/paper/2302.06706/) (2023)
- [Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings](https://mlsanity.com/paper/1704.07130/) (2017)
- [The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue](https://mlsanity.com/paper/1906.01530/) (2019)
- [A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context](https://mlsanity.com/paper/1907.03399/) (2019)
- [Hierarchical Text Generation and Planning for Strategic Dialogue](https://mlsanity.com/paper/1712.05846/) (2017)
- [Learning Executable Semantic Parsers for Natural Language Understanding](https://mlsanity.com/paper/1603.06677/) (2016)
- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [Reference-Centric Models for Grounded Collaborative Dialogue](https://mlsanity.com/paper/2109.05042/) (2021)
- [Learning an Executable Neural Semantic Parser](https://mlsanity.com/paper/1711.05066/) (2017)
- [Planning, Inference and Pragmatics in Sequential Language Games](https://mlsanity.com/paper/1805.11774/) (2018)

## Built on by (most cited first, 5 of 5)

- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Can LLM Graph Reasoning Generalize beyond Pattern Memorization?](https://mlsanity.com/paper/2406.15992/) (2024)
- [Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People](https://mlsanity.com/paper/2510.20886/) (2025)
- [Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft](https://mlsanity.com/paper/2406.17553/) (2024)
- [Program Synthesis Dialog Agents for Interactive Decision-Making](https://mlsanity.com/paper/2502.19610/) (2025)

## Abstract (arXiv)

Large language models (LLMs) excel at processing and generating both text and code. However, LLMs have had limited applicability in grounded task-oriented dialogue as they are difficult to steer toward task objectives and fail to handle novel grounding. We present a modular and interpretable grounded dialogue system that addresses these shortcomings by composing LLMs with a symbolic planner and grounded code execution. Our system consists of a reader and planner: the reader leverages an LLM to convert partner utterances into executable code, calling functions that perform grounding. The translated code's output is stored to track dialogue state, while a symbolic planner determines the next appropriate response. We evaluate our system's performance on the demanding OneCommon dialogue task, involving collaborative reference resolution on abstract images of scattered dots. Our system substantially outperforms the previous state-of-the-art, including improving task success in human evaluations from 56% to 69% in the most challenging setting.