# Goal Inference from Open-Ended Dialog arXiv: https://arxiv.org/abs/2410.13957 ยท HTML: https://mlsanity.com/paper/2410.13957/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 43 of its 69 references that are in the map. - First posted on arXiv 17 Oct 2024. ## Builds on (most cited first, 30 of 43) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022) - [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023) - [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023) - [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022) - [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021) - [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022) - [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021) - [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017) - [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022) - [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023) - [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023) - [Role-Play with Large Language Models](https://mlsanity.com/paper/2305.16367/) (2023) - [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023) - [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023) - [Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings](https://mlsanity.com/paper/1803.08495/) (2018) - [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021) - [CommonsenseQA 2.0: Exposing the Limits of AI through Gamification](https://mlsanity.com/paper/2201.05320/) (2022) - [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023) - [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018) - [Environment-Independent Task Specifications via GLTL](https://mlsanity.com/paper/1704.04341/) (2017) - [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024) - [Bayesian Preference Elicitation with Language Models](https://mlsanity.com/paper/2403.05534/) (2024) - [Learning Preferences for Manipulation Tasks from Online Coactive Feedback](https://mlsanity.com/paper/1601.00741/) (2016) ## Built on by (most cited first, 4 of 4) - [MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation](https://mlsanity.com/paper/2502.11903/) (2025) - [Open-Ended Goal Inference through Actions and Language for Human-Robot Collaboration](https://mlsanity.com/paper/2512.04453/) (2025) - [Intentionality is a Design Decision: Measuring Functional Intentionality for Accountable AI Systems](https://mlsanity.com/paper/2605.05475/) (2026) - [Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration](https://mlsanity.com/paper/2609.13982/) (2026) ## Abstract (arXiv) Embodied AI Agents are quickly becoming important and common tools in society. These embodied agents should be able to learn about and accomplish a wide range of user goals and preferences efficiently and robustly. Large Language Models (LLMs) are often used as they allow for opportunities for rich and open-ended dialog type interaction between the human and agent to accomplish tasks according to human preferences. In this thesis, we argue that for embodied agents that deal with open-ended dialog during task assistance: 1) AI Agents should extract goals from conversations in the form of Natural Language (NL) to be better at capturing human preferences as it is intuitive for humans to communicate their preferences on tasks to agents through natural language. 2) AI Agents should quantify/maintain uncertainty about these goals to ensure that actions are being taken according to goals that the agent is extremely certain about. We present an online method for embodied agents to learn and accomplish diverse user goals. While offline methods like RLHF can represent various goals but require large datasets, our approach achieves similar flexibility with online efficiency. We extract natural language goal representations from conversations with Large Language Models (LLMs). We prompt an LLM to role play as a human with different goals and use the corresponding likelihoods to run Bayesian inference over potential goals. As a result, our method can represent uncertainty over complex goals based on unrestricted dialog. We evaluate in a text-based grocery shopping domain and an AI2Thor robot simulation. We compare our method to ablation baselines that lack either explicit goal representation or probabilistic inference.