mlsanityData to 7 Oct 2026

What are you optimizing for? Aligning Recommender Systems with Human Values

arXiv 2107.10939 ↗First posted 22 Jul 2021cs.IR cs.CY cs.LGcited by 28

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

We describe cases where real recommender systems were modified in the service of various human values such as diversity, fairness, well-being, time well spent, and factual accuracy. From this we identify the current practice of values engineering: the creation of classifiers from human-created data with value-based labels. This has worked in practice for a variety of issues, but problems are addressed one at a time, and users and other stakeholders have seldom been involved. Instead, we look to AI alignment work for approaches that could learn complex values directly from stakeholders, and identify four major directions: useful measures of alignment, participatory design and operation, interactive value learning, and informed deliberative judgments.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 59 references that are in the map. First posted on arXiv 22 Jul 2021.

Builds on

14 of 59 references.

References · 14

  1. 2020Artificial Intelligence, Values and Alignmentcited by 443
  2. 2019Tackling Climate Change with Machine Learningcited by 264
  3. 2017Inverse Reward Designcited by 246
  4. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  5. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  6. 2018RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertisingcited by 65
  7. 2018Incomplete Contracting and AI Alignmentcited by 60
  8. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  9. 2018POTs: Protective Optimization Technologiescited by 28
  10. 2018Simplifying Reward Design through Divide-and-Conquercited by 12
  11. 2020IEEE 7010: A New Standard for Assessing the Well-being Implications of Artificial Intelligencecited by 11
  12. 2020Beyond Optimizing for Clicks: Incorporating Editorial Values in News Recommendationcited by 8
2 more
  1. 2019Beyond Personalization: Research Directions in Multistakeholder Recommendationcited by 4
  2. 2019Aligning Daily Activities with Personality: Towards A Recommender System for Improving Wellbeingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 4; 2024: 6; 2025: 5; 2026: 3.

28 papers.

Sort:

By area:

  1. 2021Unsolved Problems in ML Safetycited by 291
  2. 2023Large Language Model Alignment: A Surveycited by 241
  3. 2023An Overview of Catastrophic AI Riskscited by 214
  4. 2022Discovering Agentscited by 28
  5. 2022Preference Dynamics Under Personalized Recommendationscited by 25
  6. 2021User Tampering in Reinforcement Learning Recommender Systemscited by 23
  7. 2024Clickbait vs. Quality: How Engagement-Based Optimization Shapes the Content Landscape in Online Platformscited by 13
  8. 2022How Would The Viewer Feel? Estimating Wellbeing From Video Scenarioscited by 9
  9. 2025Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Valuecited by 6
  10. 2022Fast online ranking with fairness of exposurecited by 5
  11. 2024Can Probabilistic Feedback Drive User Impacts in Online Platforms?cited by 5
  12. 2024Impact of Decentralized Learning on Player Utilities in Stackelberg Gamescited by 5
16 more
  1. 2024Recommender Systems for Sustainability: Overview and Research Issuescited by 5
  2. 2022Quantitative AI Risk Assessments: Opportunities and Challengescited by 4
  3. 2022The dangers in algorithms learning humans' values and irrationalitiescited by 3
  4. 2023The Value-Sensitive Conversational Agent Co-Design Frameworkcited by 3
  5. 2022Contextual bandits with concave rewards, and an application to fair rankingcited by 2
  6. 2024Augmenting Offline Reinforcement Learning with State-only Interactionscited by 2
  7. 2024System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processescited by 2
  8. 2025Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignmentcited by 2
  9. 2025Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Taskscited by 1
  10. 2025Beyond Algorethics: Addressing the Ethical and Anthropological Challenges of AI Recommender Systemscited by 1
  11. 2022Modelling the Recommender Alignment Problemcited by 0
  12. 2023Dynamic value alignment through preference aggregation of multiple objectivescited by 0
  13. 2025MultiScale Contextual Bandits for Long Term Objectivescited by 0
  14. 2026Transparent and Controllable Recommendation Filtering via Multimodal Multi-Agent Collaborationcited by 0
  15. 2026Normative Alignment of Recommender Systems via Internal Label Shiftcited by 0
  16. 2026Toward Personal Intelligence Through Cooperative Observationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Measuring Value Alignment
  2. 2026Toward a Theory of Value in AI Alignment
  3. 2024ValueCompass: A Framework for Measuring Contextual Value Alignment Between Human and LLMs
  4. 2024What are human values, and how do we align AI to them?
  5. 2024Democratizing Reward Design for Personal and Representative Value-Alignment
  6. 2024MAP: Multi-Human-Value Alignment Palette
  7. 2023A Multi-Level Framework for the AI Alignment Problem
  8. 2025Value Alignment of Social Media Ranking Algorithms