mlsanityData to 7 Oct 2026

Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences

arXiv 2312.09337 ↗First posted 14 Dec 2023cs.CV cs.AI cs.ROcited by 25

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Customizing robotic behaviors to be aligned with diverse human preferences is an underexplored challenge in the field of embodied AI. In this paper, we present Promptable Behaviors, a novel framework that facilitates efficient personalization of robotic agents to diverse human preferences in complex environments. We use multi-objective reinforcement learning to train a single policy adaptable to a broad spectrum of preferences. We introduce three distinct methods to infer human preferences by leveraging different types of interactions: (1) human demonstrations, (2) preference feedback on trajectory comparisons, and (3) language instructions. We evaluate the proposed method in personalized object-goal navigation and flee navigation tasks in ProcTHOR and RoboTHOR, demonstrating the ability to prompt agent behaviors to satisfy human preferences in various scenarios. Project page: https://promptable-behaviors.github.io

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 79 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

39 of 79 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Function24 May
  3. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  4. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  5. 2021Learning Multimodal Rewards from Rankings27 Sep
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  9. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  10. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  11. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  12. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
20 more
  1. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  2. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  3. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
  4. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  5. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  6. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  7. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  8. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  9. 2016Multi-Objective Deep Reinforcement Learningcited by 81
  10. 2020Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewardscited by 60
  11. 2020AllenAct: A Framework for Embodied AI Researchcited by 54
  12. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
  13. 2017Multi-objective Bandits: Optimizing the Generalized Gini Indexcited by 27
  14. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 24
  15. 2022Pareto Conditioned Networkscited by 22
  16. 2023Selective Visual Representations Improve Convergence and Generalization for Embodied AIcited by 19
  17. 2022Ask4Help: Learning to Leverage an Expert for Embodied Taskscited by 18
  18. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  19. 2019The Green Choice: Learning and Influencing Human Decisions on Shared Roadscited by 14
  20. 2021Navigation In Urban Environments Amongst Pedestrians Using Multi-Objective Deep Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 8; 2026: 7.

25 papers.

Sort:

By area:

  1. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  2. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  3. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  4. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  5. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 21
  6. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 13
  7. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  8. 2026TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Ratecited by 6
  9. 2024How to Find the Exact Pareto Front for Multi-Objective MDPs?cited by 5
  10. 2024MotIF: Motion Instruction Fine-tuningcited by 4
  11. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  12. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
13 more
  1. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  2. 2025Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Languagecited by 4
  3. 2024Comparison-based Active Preference Learning for Multi-dimensional Personalizationcited by 3
  4. 2025LLM Agents for Bargaining with Utility-based Feedbackcited by 3
  5. 2025Optimal Interactive Learning on the Job via Facility Location Planningcited by 2
  6. 2026MERIT Feedback Elicits Better Bargaining in LLM Negotiatorscited by 2
  7. 2026PSG-Nav: Probabilistic Scene Graph Navigation via Multiverse Decision Makingcited by 2
  8. 2024Inferring Preferences from Demonstrations in Multi-objective Reinforcement Learningcited by 1
  9. 2025QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignmentcited by 1
  10. 2026Freeform Preference Learning for Robotic Manipulationcited by 1
  11. 2026Generating Robust Portfolios of Optimization Models using Large Language Modelscited by 0
  12. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimizationcited by 0
  13. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
  2. 2026Learning Personalized Agents from Human Feedback
  3. 2025Efficiently Generating Expressive Quadruped Behaviors via Language-Guided Preference Learning
  4. 2022Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method
  5. 2025Inferring the Unspoken: Aligning Embodied Agents with Implicit Preferences
  6. 2025Predictive Preference Learning from Human Interventions
  7. 2026From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning
  8. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferences