mlsanityData to 14 Oct 2026

Value Internalization: Learning and Generalizing from Social Reward

arXiv 2407.14681 ↗First posted 19 Jul 2024cs.LG cs.AI cs.MAcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Social rewards shape human behavior. During development, a caregiver guides a learner's behavior towards culturally aligned goals and values. How do these behaviors persist and generalize when the caregiver is no longer present, and the learner must continue autonomously? Here, we propose a model of value internalization where social feedback trains an internal social reward (ISR) model that generates internal rewards when social rewards are unavailable. Through empirical simulations, we show that an ISR model prevents agents from unlearning socialized behaviors and enables generalization in out-of-distribution tasks. We characterize the implications of incomplete internalization, akin to "reward hacking" on the ISR. Additionally, we show that our model internalizes prosocial behavior in a multi-agent environment. Our work provides a foundation for understanding how humans acquire and generalize values and offers insights for aligning AI with human values.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 51 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

11 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  4. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  5. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  6. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  7. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  8. 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
  9. 2017Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physicscited by 58

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Self-Interest: Modeling Social-Oriented Motivation for Human-like Multi-Agent Interactions
  2. 2023Generalization Analogies: A Testbed for Generalizing AI Oversight to Hard-To-Measure Domains
  3. 2026Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents
  4. 2026Self-Referenced Social Preferences: Cooperation without Observing Others Rewards
  5. 2025Reflective Verbal Reward Design for Pluralistic Alignment
  6. 2023Culturally-Attuned Moral Machines: Implicit Learning of Human Value Systems by AI through Inverse Reinforcement Learning
  7. 2025Inference of Altruism and Intrinsic Rewards in Multi-Agent Systems ↗
  8. 2026Calibrating Artificial Guilt: Neurally Grounded Reward Shaping for Prosocial Multi-Agent Reinforcement Learning