# Research Priorities for Robust and Beneficial Artificial Intelligence

arXiv: https://arxiv.org/abs/1602.03506 · HTML: https://mlsanity.com/paper/1602.03506/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2016.

## Built on by (most cited first, 30 of 74)

- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)
- [Plex: Towards Reliability using Pretrained Large Model Extensions](https://mlsanity.com/paper/2207.07411/) (2022)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [Melting Pot 2.0](https://mlsanity.com/paper/2211.13746/) (2022)
- [Transitioning to human interaction with AI systems: New challenges and opportunities for HCI professionals to enable human-centered AI](https://mlsanity.com/paper/2105.05424/) (2021)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architectures](https://mlsanity.com/paper/2310.03659/) (2023)
- [Interactive AI Alignment: Specification, Process, and Evaluation Alignment](https://mlsanity.com/paper/2311.00710/) (2023)
- [On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models](https://mlsanity.com/paper/2403.04204/) (2024)
- [Learning Norms from Stories: A Prior for Value Aligned Agents](https://mlsanity.com/paper/1912.03553/) (2019)
- [Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests](https://mlsanity.com/paper/2507.05619/) (2025)
- [Training Value-Aligned Reinforcement Learning Agents Using a Normative Prior](https://mlsanity.com/paper/2104.09469/) (2021)
- [Contestable AI needs Computational Argumentation](https://mlsanity.com/paper/2405.10729/) (2024)
- [Towards a multi-stakeholder value-based assessment framework for algorithmic systems](https://mlsanity.com/paper/2205.04525/) (2022)
- [Episodic memory in AI agents poses risks that should be studied and mitigated](https://mlsanity.com/paper/2501.11739/) (2025)
- [Human Control: Definitions and Algorithms](https://mlsanity.com/paper/2305.19861/) (2023)
- [Auditing of AI: Legal, Ethical and Technical Approaches](https://mlsanity.com/paper/2407.06235/) (2024)
- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'](https://mlsanity.com/paper/2505.04950/) (2025)
- [Non-Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1911.04220/) (2019)
- [Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations](https://mlsanity.com/paper/2109.11018/) (2021)
- [Model-Free Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.02907/) (2021)
- [Value Preferences Estimation and Disambiguation in Hybrid Participatory Systems](https://mlsanity.com/paper/2402.16751/) (2024)
- [Sociotechnical Implications of Generative Artificial Intelligence for Information Access](https://mlsanity.com/paper/2405.11612/) (2024)
- [On managing vulnerabilities in AI/ML systems](https://mlsanity.com/paper/2101.10865/) (2021)
- [Axes for Sociotechnical Inquiry in AI Research](https://mlsanity.com/paper/2105.06551/) (2021)
- [When Is It Acceptable to Break the Rules? Knowledge Representation of Moral Judgement Based on Empirical Data](https://mlsanity.com/paper/2201.07763/) (2022)
- [A Strong Baseline for Batch Imitation Learning](https://mlsanity.com/paper/2302.02788/) (2023)

## Abstract (arXiv)

Success in the quest for artificial intelligence has the potential to bring unprecedented benefits to humanity, and it is therefore worthwhile to investigate how to maximize these benefits while avoiding potential pitfalls. This article gives numerous examples (which should by no means be construed as an exhaustive list) of such worthwhile research aimed at ensuring that AI remains robust and beneficial.