Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
When applying reinforcement learning (RL) to a new problem, reward engineering is a necessary, but often difficult and error-prone task a system designer has to face. To avoid this step, we propose LR4GPM, a novel (deep) RL method that can optimize a global performance metric, which is supposed to be available as part of the problem description. LR4GPM alternates between two phases: (1) learning a (possibly vector) reward function used to fit the performance metric, and (2) training a policy to optimize an approximation of this performance metric based on the learned rewards. Such RL training is not straightforward since both the reward function and the policy are trained using non-stationary data. To overcome this issue, we propose several training tricks. We demonstrate the efficiency of LR4GPM on several domains. Notably, LR4GPM outperforms the winner of a recent autonomous driving competition organized at DAI'2020.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 57 references that are in the map. First posted on arXiv 16 Mar 2023.
Builds on
23 of 57 references.
References · 23
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
- 2016OpenAI Gymcited by 1,684
- 2016Concrete Problems in AI Safetycited by 1,582
- 2017Constrained Policy Optimizationcited by 928
- 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
- 2018Reward learning from human preferences and demonstrations in Ataricited by 314
- 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
- 2020First return, then explorecited by 256
11 more
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
- 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
- 2015Policy Gradient for Coherent Risk Measurescited by 74
- 2021On the Expressivity of Markov Rewardcited by 71
- 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
- 2020Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewardscited by 60
- 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
- 2020Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learningcited by 40
- 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
- 2021Safe Distributional Reinforcement Learningcited by 10
- 2018Iroko: A Framework to Prototype Reinforcement Learning for Data Center Traffic Controlcited by 3
Built on by
Citing papers per year: 2023: 1.
1 paper.