# Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences

arXiv: https://arxiv.org/abs/2312.09337 · HTML: https://mlsanity.com/paper/2312.09337/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 39 of its 79 references that are in the map.
- First posted on arXiv 14 Dec 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [SOON: Scenario Oriented Object Navigation with Graph-based Exploration](https://mlsanity.com/paper/2103.17138/) (2021)
- [Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale](https://mlsanity.com/paper/2204.03514/) (2022)
- [SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning](https://mlsanity.com/paper/2203.10050/) (2022)
- [PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav](https://mlsanity.com/paper/2301.07302/) (2023)
- [Batch Active Preference-Based Learning of Reward Functions](https://mlsanity.com/paper/1810.04303/) (2018)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/1610.02707/) (2016)
- [Contrastive Preference Learning: Learning from Human Feedback without RL](https://mlsanity.com/paper/2310.13639/) (2023)
- [Inverse Preference Learning: Preference-based RL without a Reward Function](https://mlsanity.com/paper/2305.15363/) (2023)
- [Reward Uncertainty for Exploration in Preference-based Reinforcement Learning](https://mlsanity.com/paper/2205.12401/) (2022)
- [Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewards](https://mlsanity.com/paper/2008.07773/) (2020)
- [AllenAct: A Framework for Embodied AI Research](https://mlsanity.com/paper/2008.12760/) (2020)
- [Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization](https://mlsanity.com/paper/2301.07784/) (2023)

## Built on by (most cited first, 26 of 26)

- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [Panacea: Pareto Alignment via Preference Adaptation for LLMs](https://mlsanity.com/paper/2402.02030/) (2024)
- [RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation](https://mlsanity.com/paper/2405.00254/) (2024)
- [Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning](https://mlsanity.com/paper/2407.15762/) (2024)
- [Robust Multi-Objective Preference Alignment with Online DPO](https://mlsanity.com/paper/2503.00295/) (2025)
- [Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2407.17466/) (2024)
- [TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate](https://mlsanity.com/paper/2605.13909/) (2026)
- [Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2405.00746/) (2024)
- [How to Find the Exact Pareto Front for Multi-Objective MDPs?](https://mlsanity.com/paper/2410.15557/) (2024)
- [MotIF: Motion Instruction Fine-tuning](https://mlsanity.com/paper/2409.10683/) (2024)
- [A Shared Low-Rank Adaptation Approach to Personalized RLHF](https://mlsanity.com/paper/2503.19201/) (2025)
- [LLM Agents for Bargaining with Utility-based Feedback](https://mlsanity.com/paper/2505.22998/) (2025)
- [Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2507.04340/) (2025)
- [PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agents](https://mlsanity.com/paper/2509.19843/) (2025)
- [Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Language](https://mlsanity.com/paper/2511.14565/) (2025)
- [Comparison-based Active Preference Learning for Multi-dimensional Personalization](https://mlsanity.com/paper/2411.00524/) (2024)
- [Demonstration-Enhanced Adaptable Multi-Objective Robot Navigation](https://mlsanity.com/paper/2404.04857/) (2024)
- [QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignment](https://mlsanity.com/paper/2503.05035/) (2025)
- [Optimal Interactive Learning on the Job via Facility Location Planning](https://mlsanity.com/paper/2505.00490/) (2025)
- [MERIT Feedback Elicits Better Bargaining in LLM Negotiators](https://mlsanity.com/paper/2602.10467/) (2026)
- [PSG-Nav: Probabilistic Scene Graph Navigation via Multiverse Decision Making](https://mlsanity.com/paper/2606.01313/) (2026)
- [Inferring Preferences from Demonstrations in Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2409.20258/) (2024)
- [Freeform Preference Learning for Robotic Manipulation](https://mlsanity.com/paper/2606.32027/) (2026)
- [Generating Robust Portfolios of Optimization Models using Large Language Models](https://mlsanity.com/paper/2605.27013/) (2026)
- [To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization](https://mlsanity.com/paper/2608.18770/) (2026)
- [The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization](https://mlsanity.com/paper/2609.39055/) (2026)

## Abstract (arXiv)

Customizing robotic behaviors to be aligned with diverse human preferences is an underexplored challenge in the field of embodied AI. In this paper, we present Promptable Behaviors, a novel framework that facilitates efficient personalization of robotic agents to diverse human preferences in complex environments. We use multi-objective reinforcement learning to train a single policy adaptable to a broad spectrum of preferences. We introduce three distinct methods to infer human preferences by leveraging different types of interactions: (1) human demonstrations, (2) preference feedback on trajectory comparisons, and (3) language instructions. We evaluate the proposed method in personalized object-goal navigation and flee navigation tasks in ProcTHOR and RoboTHOR, demonstrating the ability to prompt agent behaviors to satisfy human preferences in various scenarios. Project page: https://promptable-behaviors.github.io