# What are you optimizing for? Aligning Recommender Systems with Human Values

arXiv: https://arxiv.org/abs/2107.10939 · HTML: https://mlsanity.com/paper/2107.10939/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 14 of its 59 references that are in the map.
- First posted on arXiv 22 Jul 2021.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 14 of 14)

- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Active Preference-Based Gaussian Process Regression for Reward Learning](https://mlsanity.com/paper/2005.02575/) (2020)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)
- [The Problem with Metrics is a Fundamental Problem for AI](https://mlsanity.com/paper/2002.08512/) (2020)
- [POTs: Protective Optimization Technologies](https://mlsanity.com/paper/1806.02711/) (2018)
- [Simplifying Reward Design through Divide-and-Conquer](https://mlsanity.com/paper/1806.02501/) (2018)
- [IEEE 7010: A New Standard for Assessing the Well-being Implications of Artificial Intelligence](https://mlsanity.com/paper/2005.06620/) (2020)
- [Beyond Optimizing for Clicks: Incorporating Editorial Values in News Recommendation](https://mlsanity.com/paper/2004.09980/) (2020)
- [Beyond Personalization: Research Directions in Multistakeholder Recommendation](https://mlsanity.com/paper/1905.01986/) (2019)
- [Aligning Daily Activities with Personality: Towards A Recommender System for Improving Wellbeing](https://mlsanity.com/paper/1909.03847/) (2019)

## Built on by (most cited first, 29 of 29)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Large Language Model Safety: A Holistic Survey](https://mlsanity.com/paper/2412.17686/) (2024)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [Preference Dynamics Under Personalized Recommendations](https://mlsanity.com/paper/2205.13026/) (2022)
- [User Tampering in Reinforcement Learning Recommender Systems](https://mlsanity.com/paper/2109.04083/) (2021)
- [Clickbait vs. Quality: How Engagement-Based Optimization Shapes the Content Landscape in Online Platforms](https://mlsanity.com/paper/2401.09804/) (2024)
- [How Would The Viewer Feel? Estimating Wellbeing From Video Scenarios](https://mlsanity.com/paper/2210.10039/) (2022)
- [Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value](https://mlsanity.com/paper/2512.03399/) (2025)
- [Fast online ranking with fairness of exposure](https://mlsanity.com/paper/2209.13019/) (2022)
- [Can Probabilistic Feedback Drive User Impacts in Online Platforms?](https://mlsanity.com/paper/2401.05304/) (2024)
- [Impact of Decentralized Learning on Player Utilities in Stackelberg Games](https://mlsanity.com/paper/2403.00188/) (2024)
- [Recommender Systems for Sustainability: Overview and Research Issues](https://mlsanity.com/paper/2412.03620/) (2024)
- [Quantitative AI Risk Assessments: Opportunities and Challenges](https://mlsanity.com/paper/2209.06317/) (2022)
- [The dangers in algorithms learning humans' values and irrationalities](https://mlsanity.com/paper/2202.13985/) (2022)
- [The Value-Sensitive Conversational Agent Co-Design Framework](https://mlsanity.com/paper/2310.11848/) (2023)
- [Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignment](https://mlsanity.com/paper/2509.04445/) (2025)
- [Contextual bandits with concave rewards, and an application to fair ranking](https://mlsanity.com/paper/2210.09957/) (2022)
- [Augmenting Offline Reinforcement Learning with State-only Interactions](https://mlsanity.com/paper/2402.00807/) (2024)
- [System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processes](https://mlsanity.com/paper/2406.01611/) (2024)
- [Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks](https://mlsanity.com/paper/2507.02819/) (2025)
- [Beyond Algorethics: Addressing the Ethical and Anthropological Challenges of AI Recommender Systems](https://mlsanity.com/paper/2507.16430/) (2025)
- [Modelling the Recommender Alignment Problem](https://mlsanity.com/paper/2208.12299/) (2022)
- [Dynamic value alignment through preference aggregation of multiple objectives](https://mlsanity.com/paper/2310.05871/) (2023)
- [MultiScale Contextual Bandits for Long Term Objectives](https://mlsanity.com/paper/2503.17674/) (2025)
- [Transparent and Controllable Recommendation Filtering via Multimodal Multi-Agent Collaboration](https://mlsanity.com/paper/2604.17459/) (2026)
- [Normative Alignment of Recommender Systems via Internal Label Shift](https://mlsanity.com/paper/2607.10915/) (2026)
- [Toward Personal Intelligence Through Cooperative Observation](https://mlsanity.com/paper/2608.17128/) (2026)

## Abstract (arXiv)

We describe cases where real recommender systems were modified in the service of various human values such as diversity, fairness, well-being, time well spent, and factual accuracy. From this we identify the current practice of values engineering: the creation of classifiers from human-created data with value-based labels. This has worked in practice for a variety of issues, but problems are addressed one at a time, and users and other stakeholders have seldom been involved. Instead, we look to AI alignment work for approaches that could learn complex values directly from stakeholders, and identify four major directions: useful measures of alignment, participatory design and operation, interactive value learning, and informed deliberative judgments.