# Learning Transferable Latent User Preferences for Human-Aligned Decision Making

arXiv: https://arxiv.org/abs/2605.12682 · HTML: https://mlsanity.com/paper/2605.12682/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 12 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 32)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning](https://mlsanity.com/paper/2307.06135/) (2023)
- [Preference Ranking Optimization for Human Alignment](https://mlsanity.com/paper/2306.17492/) (2023)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Text2Reward: Reward Shaping with Language Models for Reinforcement Learning](https://mlsanity.com/paper/2309.11489/) (2023)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Task and Motion Planning with Large Language Models for Object Rearrangement](https://mlsanity.com/paper/2303.06247/) (2023)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Panacea: Pareto Alignment via Preference Adaptation for LLMs](https://mlsanity.com/paper/2402.02030/) (2024)
- [Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents](https://mlsanity.com/paper/2309.09919/) (2023)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [Group Preference Optimization: Few-Shot Alignment of Large Language Models](https://mlsanity.com/paper/2310.11523/) (2023)
- [Housekeep: Tidying Virtual Households using Commonsense Reasoning](https://mlsanity.com/paper/2205.10712/) (2022)
- [Understanding Large-Language Model (LLM)-powered Human-Robot Interaction](https://mlsanity.com/paper/2401.03217/) (2024)
- [InterPreT: Interactive Predicate Learning from Language Feedback for Generalizable Task Planning](https://mlsanity.com/paper/2405.19758/) (2024)
- [Large Language Models as Zero-Shot Human Models for Human-Robot Interaction](https://mlsanity.com/paper/2303.03548/) (2023)
- [Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity](https://mlsanity.com/paper/2402.06529/) (2024)
- [Generative Expressive Robot Behaviors using Large Language Models](https://mlsanity.com/paper/2401.14673/) (2024)
- [Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Models](https://mlsanity.com/paper/2309.04316/) (2023)
- [LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robots](https://mlsanity.com/paper/2404.14285/) (2024)
- [Task Supportive and Personalized Human-Large Language Model Interaction: A User Study](https://mlsanity.com/paper/2402.06170/) (2024)
- [AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment](https://mlsanity.com/paper/2506.04089/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used as reasoning modules in many applications. While they are efficient in certain tasks, LLMs often struggle to produce human-aligned solutions. Human-aligned decision making requires accounting for both explicitly stated goals and latent user preferences that shape how ambiguous situations should be resolved. Existing approaches to incorporating such preferences either rely on extensive and repeated user interactions or fail to generalize latent preferences across tasks and contexts, limiting their practical applicability. We consider a setting in which an LLM is used for high-level reasoning and is responsible for inferring latent user preferences from limited interactions, which guides downstream decision making. We introduce CLIPR (Conversational Learning for Inferring Preferences and Reasoning), a framework that learns actionable, transferable natural language rules that represent latent user preferences from minimal conversational input. These rules are iteratively refined through adaptive feedback and applied to both in-distribution and out-of-distribution ambiguous tasks across multiple environments. Evaluations on three datasets and a user study show that CLIPR consistently outperforms existing methods in improving alignment and reducing inference costs.