mlsanityData to 14 Oct 2026

Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards

arXiv 2310.00435 ↗First posted 30 Sep 2023cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, it is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian with respect to the individual objectives. To this end, a practical non-Markovian aggregation scheme is proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 35 of its 88 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

35 of 88 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  8. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  9. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  10. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  11. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  12. 2018Dynamic Weights in Multi-Objective Deep Reinforcement Learningcited by 100
23 more
  1. 2016The Predictron: End-To-End Learning and Planningcited by 93
  2. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 72
  3. 2021On the Expressivity of Markov Rewardcited by 71
  4. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 60
  5. 2021Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)cited by 55
  6. 2020A Boolean Task Algebra for Reinforcement Learningcited by 48
  7. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  8. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 45
  9. 2016Unifying task specification in reinforcement learningcited by 41
  10. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  11. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  12. 2022Settling the Reward Hypothesiscited by 33
  13. 2019Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approachcited by 32
  14. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  15. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 26
  16. 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
  17. 2020Online Learning of Non-Markovian Reward Modelscited by 9
  18. 2022Utility Theory for Sequential Decision Makingcited by 8
  19. 2017Multi-Advisor Reinforcement Learningcited by 7
  20. 2022Reinforcement Learning with Non-Exponential Discountingcited by 7
  21. 2022(Im)possibility of Collective Intelligencecited by 4
  22. 2023On the Expressivity of Multidimensional Markov Rewardcited by 4
  23. 2023The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 3; 2026: 2.

8 papers.

  1. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  2. 2025Benchmarking and Improving LLM Robustness for Personalized Generationcited by 4
  3. 2024A Black Swan Hypothesis: The Role of Human Irrationality in AI Safetycited by 3
  4. 2024Pluralistic Alignment Over Timecited by 3
  5. 2025Dual-Objective Reinforcement Learning with Novel Hamilton-Jacobi-Bellman Formulationscited by 3
  6. 2025The challenge of hidden gifts in multi-agent reinforcement learningcited by 1
  7. 2026Rationalizing Boltzmann Rationality: An Axiomatic Characterization of Entropy-Regularized Policiescited by 0
  8. 2026Fragility of Value under Imperfect Alignmentcited by 0