mlsanityData to 7 Oct 2026

MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning

arXiv 2201.00012 ↗First posted 30 Dec 2021cs.LG cs.AIcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Inferring reward functions from demonstrations and pairwise preferences are auspicious approaches for aligning Reinforcement Learning (RL) agents with human intentions. However, state-of-the art methods typically focus on learning a single reward model, thus rendering it difficult to trade off different reward functions from multiple experts. We propose Multi-Objective Reinforced Active Learning (MORAL), a novel method for combining diverse demonstrations of social norms into a Pareto-optimal policy. Through maintaining a distribution over scalarization weights, our approach is able to interactively tune a deep RL agent towards a variety of preferences, while eliminating the need for computing multiple policies. We empirically demonstrate the effectiveness of MORAL in two scenarios, which model a delivery and an emergency task that require an agent to act in the presence of normative conflicts. Overall, we consider our research a step towards multi-objective RL with learned rewards, bridging the gap between current reward learning and machine ethics literature.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 55 references that are in the map. First posted on arXiv 30 Dec 2021.

Builds on

31 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Learning Norms from Stories: A Prior for Value Aligned Agents7 Dec

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  5. 2020Artificial Intelligence, Values and Alignmentcited by 443
  6. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  7. 2016Cooperative Inverse Reinforcement Learningcited by 298
  8. 2019Challenges of Real-World Reinforcement Learningcited by 282
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  10. 2017Inverse Reward Designcited by 246
  11. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  12. 2019A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptationcited by 166
18 more
  1. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  2. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  3. 2016Research Priorities for Robust and Beneficial Artificial Intelligencecited by 74
  4. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  5. 2020Thinking Fast and Slow in AIcited by 67
  6. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  7. 2018Incomplete Contracting and AI Alignmentcited by 60
  8. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 51
  9. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
  10. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  11. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Informationcited by 41
  12. 2021Learning Multimodal Rewards from Rankingscited by 41
  13. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learningcited by 36
  14. 2018Multi-task Maximum Entropy Inverse Reinforcement Learningcited by 20
  15. 2017A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agentscited by 18
  16. 2020Reinforcement Learning Under Moral Uncertaintycited by 16
  17. 2021Combining Reward Information from Multiple Sourcescited by 7
  18. 2020oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 11; 2025: 4; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  3. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  4. 2023Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learningcited by 10
  5. 2023Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifestocited by 7
  6. 2024Adaptive Alignment: Dynamic Preference Adjustments via Multi-Objective Reinforcement Learning for Pluralistic AIcited by 6
  7. 2024MOMAland: A Set of Benchmarks for Multi-Objective Multi-Agent Reinforcement Learningcited by 5
  8. 2024How to Find the Exact Pareto Front for Multi-Objective MDPs?cited by 5
  9. 2023Factual and Personalized Recommendations using Language Models and Reinforcement Learningcited by 4
  10. 2024Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignmentcited by 4
  11. 2024Scalable Multi-Objective Reinforcement Learning with Fairness Guarantees using Lorenz Dominancecited by 4
  12. 2024Reinforcement Learning and Machine ethics:a systematic reviewcited by 3
12 more
  1. 2024ARMCHAIR: integrated inverse reinforcement learning and model predictive control for human-robot collaborationcited by 2
  2. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  3. 2025Learning the Value Systems of Societies from Preferencescited by 2
  4. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  5. 2025Beyond Algorethics: Addressing the Ethical and Anthropological Challenges of AI Recommender Systemscited by 1
  6. 2026Freeform Preference Learning for Robotic Manipulationcited by 1
  7. 2024Building Altruistic and Moral AI Agent with Brain-inspired Emotional Empathy Mechanismscited by 0
  8. 2025Efficient Portfolio Selection through Preference Aggregation with Quicksort and the Bradley--Terry Modelcited by 0
  9. 2025DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Valuescited by 0
  10. 2026Fog of Love: Engineering Virtuous Agent Behavior with Affinity-based Reinforcement Learning in a Game Environmentcited by 0
  11. 2026LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedbackcited by 0
  12. 2026Training and Evaluating Ethical Reinforcement Learning Agents on Per-Episode Distributionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implications
  2. 2024Moral Alignment for LLM Agents
  3. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  4. 2025MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning
  5. 2025Learning Pareto-Optimal Rewards from Noisy Preferences: A Framework for Multi-Objective Inverse Reinforcement Learning
  6. 2018Scalable agent alignment via reward modeling: a research direction
  7. 2023Culturally-Attuned Moral Machines: Implicit Learning of Human Value Systems by AI through Inverse Reinforcement Learning
  8. 2024Pareto-Optimal Learning from Preferences with Hidden Context