mlsanityData to 14 Oct 2026

Settling the Reward Hypothesis

arXiv 2212.10420 ↗First posted 20 Dec 2022cs.AI cs.LG math.STcited by 33

Not on the map. Why?

Abstract (arXiv)

The reward hypothesis posits that, "all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward)." We aim to fully settle this hypothesis. This will not conclude with a simple affirmation or refutation, but rather specify completely the implicit requirements on goals and purposes under which the hypothesis holds.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 11 of its 53 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

11 of 53 references.

References · 11

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2021On the Expressivity of Markov Rewardcited by 71
  3. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 60
  4. 2021Reinforcement Learning, Bit by Bitcited by 59
  5. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 43
  6. 2016Unifying task specification in reinforcement learningcited by 41
  7. 2019Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approachcited by 32
  8. 2013The Sample-Complexity of General Reinforcement Learningcited by 18
  9. 2016Nonparametric General Reinforcement Learningcited by 10
  10. 2022Utility Theory for Sequential Decision Makingcited by 8
  11. 2021Abstractions of General Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 7; 2025: 13; 2026: 5.

33 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  3. 2023A Definition of Continual Reinforcement Learningcited by 101
  4. 2024Beyond Preferences in AI Alignmentcited by 49
  5. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 24
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  7. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 16
  8. 2025Rethinking the Foundations for Continual Reinforcement Learningcited by 15
  9. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  10. 2024Three Dogmas of Reinforcement Learningcited by 14
  11. 2025Plasticity as the Mirror of Empowermentcited by 12
  12. 2025Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitionerscited by 10
21 more
  1. 2025BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RLcited by 10
  2. 2023Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewardscited by 8
  3. 2023On the Convergence of Bounded Agentscited by 7
  4. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  5. 2023Conditions on Preference Relations that Guarantee the Existence of Optimal Policiescited by 3
  6. 2024A Black Swan Hypothesis: The Role of Human Irrationality in AI Safetycited by 3
  7. 2025Toward Agents That Reason About Their Computationcited by 3
  8. 2025Value Under Ignorance in Universal Artificial Intelligencecited by 3
  9. 2025Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPscited by 2
  10. 2025Agent-centric learning: from external reward maximization to internal knowledge curationcited by 2
  11. 2025Generative Actor Criticcited by 2
  12. 2026Rewarding DINO: Predicting Dense Rewards with Vision Foundation Modelscited by 2
  13. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  14. 2026Calculating Mutual Information between a Reward Maximizer and its Environmentcited by 1
  15. 2024Learning telic-controllable state representationscited by 0
  16. 2025Closing the Intent-to-Behavior Gap via Fulfillment Priority Logiccited by 0
  17. 2025Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Lightcited by 0
  18. 2025Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learningcited by 0
  19. 2026Artifacts as Memory Beyond the Agent Boundarycited by 0
  20. 2026Darwin Mobile Agent: A Roadmap for Self-Evolutioncited by 0
  21. 2026The syntax and semantics of goalscited by 0