# Plan-Grounded Large Language Models for Dual Goal Conversational Settings

arXiv: https://arxiv.org/abs/2402.01053 · HTML: https://mlsanity.com/paper/2402.01053/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [Efficient Large Scale Language Modeling with Mixtures of Experts](https://mlsanity.com/paper/2112.10684/) (2021)
- [Understanding the Capabilities of Large Language Models for Automated Planning](https://mlsanity.com/paper/2305.16151/) (2023)
- [Opportunities and Challenges in Neural Dialog Tutoring](https://mlsanity.com/paper/2301.09919/) (2023)
- [Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generation](https://mlsanity.com/paper/2305.14386/) (2023)
- [Neuro-Symbolic Procedural Planning with Commonsense Prompting](https://mlsanity.com/paper/2206.02928/) (2022)
- [Alexa, Let's Work Together: Introducing the First Alexa Prize TaskBot Challenge on Conversational Task Assistance](https://mlsanity.com/paper/2209.06321/) (2022)
- [Task2Dial: A Novel Task and Dataset for Commonsense enhanced Task-based Dialogue Grounded in Documents](https://mlsanity.com/paper/2204.01061/) (2022)
- [TUTORING: Instruction-Grounded Conversational Agent for Language Learners](https://mlsanity.com/paper/2302.12623/) (2023)
- [Learning towards Selective Data Augmentation for Dialogue Generation](https://mlsanity.com/paper/2303.09719/) (2023)

## Built on by (most cited first, 8 of 8)

- [Gl'orIA -- A Generative and Open Large Language Model for Portuguese](https://mlsanity.com/paper/2402.12969/) (2024)
- [SPHERE: An Evaluation Card for Human-AI Systems](https://mlsanity.com/paper/2504.07971/) (2025)
- [Evaluating Behavioral Alignment in Conflict Dialogue: A Multi-Dimensional Comparison of LLM Agents and Humans](https://mlsanity.com/paper/2509.16394/) (2025)
- [Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants](https://mlsanity.com/paper/2410.12891/) (2024)
- [Show and Guide: Instructional-Plan Grounded Vision and Language Model](https://mlsanity.com/paper/2409.19074/) (2024)
- [Towards Ontology-Based Descriptions of Conversations with Qualitatively-Defined Concepts](https://mlsanity.com/paper/2509.04926/) (2025)
- [VIGiA: Instructional Video Guidance via Dialogue Reasoning and Retrieval](https://mlsanity.com/paper/2602.19146/) (2026)
- [Conversational Control with Ontologies for Large Language Models: A Lightweight Framework for Constrained Generation](https://mlsanity.com/paper/2604.04450/) (2026)

## Abstract (arXiv)

Training Large Language Models (LLMs) to follow user instructions has been shown to supply the LLM with ample capacity to converse fluently while being aligned with humans. Yet, it is not completely clear how an LLM can lead a plan-grounded conversation in mixed-initiative settings where instructions flow in both directions of the conversation, i.e. both the LLM and the user provide instructions to one another. In this paper, we tackle a dual goal mixed-initiative conversational setting where the LLM not only grounds the conversation on an arbitrary plan but also seeks to satisfy both a procedural plan and user instructions. The LLM is then responsible for guiding the user through the plan and, at the same time, adapting to new circumstances, answering questions, and activating safety guardrails when needed. We propose a novel LLM that grounds the dialogue on a procedural plan, can take the dialogue initiative, and enforces guardrails on the system's behavior, while also improving the LLM's responses to unexpected user behavior. Experiments in controlled settings and with real users show that the best-performing model, which we call PlanLLM, achieves a 2.1x improvement over a strong baseline. Moreover, experiments also show good generalization to unseen domains.