mlsanityData to 7 Oct 2026

Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences

arXiv 2210.15906 ↗First posted 28 Oct 2022cs.AI cs.HC cs.LGcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Generating complex behaviors that satisfy the preferences of non-expert users is a crucial requirement for AI agents. Interactive reward learning from trajectory comparisons (a.k.a. RLHF) is one way to allow non-expert users to convey complex objectives by expressing preferences over short clips of agent behaviors. Even though this parametric method can encode complex tacit knowledge present in the underlying tasks, it implicitly assumes that the human is unable to provide richer feedback than binary preference labels, leading to intolerably high feedback complexity and poor user experience. While providing a detailed symbolic closed-form specification of the objectives might be tempting, it is not always feasible even for an expert user. However, in most cases, humans are aware of how the agent should change its behavior along meaningful axes to fulfill their underlying purpose, even if they are not able to fully specify task objectives symbolically. Using this as motivation, we introduce the notion of Relative Behavioral Attributes, which allows the users to tweak the agent behavior through symbolic concepts (e.g., increasing the softness or speed of agents' movement). We propose two practical methods that can learn to model any kind of behavioral attributes from ordered behavior clips. We demonstrate the effectiveness of our methods on four tasks with nine different behavioral attributes, showing that once the attributes are learned, end users can produce desirable agent behaviors relatively effortlessly, by providing feedback just around ten times. This is over an order of magnitude less than that required by the popular learning-from-human-preferences baselines. The supplementary video and source code are available at: https://guansuns.github.io/pages/rba.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 65 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

35 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  7. 2022Human Motion Diffusion Modelcited by 1,062
  8. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  9. 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
  10. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  11. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  12. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
19 more
  1. 2022FLAME: Free-form Language-based Motion Synthesis & Editingcited by 217
  2. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  3. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  4. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  5. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  6. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  7. 2017Robust Imitation of Diverse Behaviorscited by 85
  8. 2022Learning Neuro-Symbolic Skills for Bilevel Planningcited by 55
  9. 2018PEORL: Integrating Symbolic Planning and Hierarchical Reinforcement Learning for Robust Decision-Makingcited by 51
  10. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  11. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  12. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  13. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  14. 2021Contrastively Disentangled Sequential Variational Autoencodercited by 34
  15. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  16. 2022Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversitycited by 22
  17. 2020Weakly-Supervised Reinforcement Learning for Controllable Behaviorcited by 17
  18. 2021Learning Perceptual Concepts by Bootstrapping from Human Queriescited by 12
  19. 2018Cost Functions for Robot Motion Stylecited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 6.

7 papers.

  1. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  2. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  3. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  4. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  5. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  6. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  7. 2024MAPLE: A Framework for Active Preference Learning Guided by Large Language Modelscited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
  2. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences
  3. 2026Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents
  4. 2025QuickLAP: Quick Language-Action Preference Learning for Semi-Autonomous Agents
  5. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferences
  6. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  7. 2019Learning Human Objectives by Evaluating Hypothetical Behavior
  8. 2025Behaviour Discovery and Attribution for Explainable Reinforcement Learning