mlsanityData to 7 Oct 2026

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

arXiv 2112.15422 ↗First posted 25 Nov 2021cs.AIcited by 55

Not on the map. Why?

Abstract (arXiv)

The recent paper `"Reward is Enough" by Silver, Singh, Precup and Sutton posits that the concept of reward maximisation is sufficient to underpin all intelligence, both natural and artificial. We contest the underlying assumption of Silver et al. that such reward can be scalar-valued. In this paper we explain why scalar rewards are insufficient to account for some aspects of both biological and computational intelligence, and argue in favour of explicitly multi-objective models of reward maximisation. Furthermore, we contend that even if scalar reward functions can trigger intelligent behaviour in specific cases, it is still undesirable to use this approach for the development of artificial general intelligence due to unacceptable risks of unsafe or unethical behaviour.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 17 of its 85 references that are in the map. First posted on arXiv 25 Nov 2021.

Builds on

17 of 85 references.

References · 17

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  3. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  4. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  5. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  6. 2017Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbolscited by 123
  7. 2017AI Safety Gridworldscited by 108
  8. 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 96
  9. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 51
  10. 2019Multi-Objective Multi-Agent Decision Making: A Utility-based Analysis and Surveycited by 45
  11. 2020Avoiding Side Effects By Considering Future Taskscited by 32
  12. 2018AGI Safety Literature Reviewcited by 28
5 more
  1. 2021Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Surveycited by 24
  2. 2021Levels of explainable artificial intelligence for human-aligned conversational explanationscited by 21
  3. 2016Superintelligence cannot be contained: Lessons from Computability Theorycited by 13
  4. 2020A utility-based analysis of equilibria in multi-objective normal form gamescited by 9
  5. 2017Identification and Off-Policy Learning of Multiple Objectives Using Adaptive Clusteringcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 7; 2024: 19; 2025: 12; 2026: 10.

55 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2024Beyond Preferences in AI Alignmentcited by 46
  3. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  4. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  5. 2023Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Frameworkcited by 28
  6. 2022Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfercited by 22
  7. 2022COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checkingcited by 19
  8. 2022On the link between conscious function and general intelligence in humans and machinescited by 15
  9. 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoningcited by 12
  10. 2022Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representationcited by 11
  11. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  12. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
38 more
  1. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  2. 2024Safety-Oriented Pruning and Interpretation of Reinforcement Learning Policiescited by 9
  3. 2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
  4. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learningcited by 8
  5. 2024Thresholded Lexicographic Ordered Multiobjective Reinforcement Learningcited by 7
  6. 2024LLMs can learn self-restraint through iterative self-reflectioncited by 6
  7. 2024Enhancing RL Safety with Counterfactual LLM Reasoningcited by 6
  8. 2024Multi-objective Deep Learning: Taxonomy and Survey of the State of the Artcited by 6
  9. 2022Targeted Adversarial Attacks on Deep Reinforcement Learning Policies via Model Checkingcited by 5
  10. 2023Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learningcited by 5
  11. 2024Probabilistic Model Checking of Stochastic Reinforcement Learning Policiescited by 5
  12. 2024Towards Aligning Language Models with Textual Feedbackcited by 5
  13. 2022Monte Carlo Tree Search Algorithms for Risk-Aware and Multi-Objective Reinforcement Learningcited by 4
  14. 2023Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safetycited by 4
  15. 2024Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learningcited by 4
  16. 2024Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignmentcited by 4
  17. 2025EMORL: Ensemble Multi-Objective Reinforcement Learning for Efficient and Flexible LLM Fine-Tuningcited by 4
  18. 2022Multi-Objective Coordination Graphs for the Expected Scalarised Returns with Generative Flow Modelscited by 3
  19. 2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3
  20. 2025Turn-based Multi-Agent Reinforcement Learning Model Checkingcited by 3
  21. 2025Increasing Interaction Fidelity: Training Routines for Biomechanical Models in HCIcited by 3
  22. 2026Beyond Rewards in Reinforcement Learning for Cyber Defencecited by 3
  23. 2026Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MCcited by 3
  24. 2023Learning Roles with Emergent Social Value Orientationscited by 2
  25. 2024An Empirical Investigation of Value-Based Multi-objective Reinforcement Learning for Stochastic Environmentscited by 2
  26. 2025Co-Activation Graph Analysis of Safety-Verified and Explainable Deep Reinforcement Learning Policiescited by 2
  27. 2025Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPscited by 2
  28. 2024From homeostasis to resource sharing: Biologically and economically aligned multi-objective multi-agent gridworld-based AI safety benchmarkscited by 1
  29. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  30. 2025BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation formatcited by 1
  31. 2024Combinatorial Client-Master Multiagent Deep Reinforcement Learning for Task Offloading in Mobile Edge Computingcited by 0
  32. 2025Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Modelscited by 0
  33. 2025Balancing Specialization and Centralization: A Multi-Agent Reinforcement Learning Benchmark for Sequential Industrial Controlcited by 0
  34. 2025MOMA-AC: A preference-driven actor-critic framework for continuous multi-objective multi-agent reinforcement learningcited by 0
  35. 2025Formal Verification of Noisy Quantum Reinforcement Learning Policiescited by 0
  36. 2026Progress Constraints for Reinforcement Learning in Behavior Treescited by 0
  37. 2026COOL-MC: Verifying and Explaining RL Policies for Platelet Inventory Managementcited by 0
  38. 2026Aligning to Illusions: Choice Blindness in Human and AI Feedbackcited by 0

50 most cited of 55 All 55 (JSON)