Data to 7 Oct 2026

On Learning Intrinsic Rewards for Policy Gradient Methods

arXiv 1804.06459 ↗First posted 17 Apr 2018cs.AI cs.LG stat.MLcited by 100

Not on the map. Why?

Abstract (arXiv)

In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem, or close variants thereof, have been proposed in the literature. In this paper we build on the Optimal Rewards Framework of Singh et.al. that defines the optimal intrinsic reward function as one that when used by an RL agent achieves behavior that optimizes the task-specifying or extrinsic reward function. Previous work in this framework has shown how good intrinsic reward functions can be learned for lookahead search based planning agents. Whether it is possible to learn intrinsic reward functions for learning agents remains an open problem. In this paper we derive a novel algorithm for learning intrinsic rewards for policy-gradient based learning agents. We compare the performance of an augmented agent that uses our algorithm to provide additive intrinsic rewards to an A2C-based policy learner (for Atari games) and a PPO-based policy learner (for Mujoco domains) with a baseline agent that uses the same policy learners but with only extrinsic rewards. Our results show improved performance on most but not all of the domains.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 25; 2022: 20; 2023: 14; 2024: 16; 2025: 15; 2026: 8.

100 papers.

Sort:

By area:

2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
2021Meta-learning PINN loss functionscited by 51
2021Bootstrapped Meta-Learningcited by 45
2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
2019MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learningcited by 29
2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
2021Discovery of Options via Meta-Learned Subgoalscited by 27
2018Generative Adversarial Self-Imitation Learningcited by 25

38 more

2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
2022Redeeming Intrinsic Rewards via Constrained Optimizationcited by 24
2022Continuously Discovering Novel Strategies via Reward-Switching Policy Optimizationcited by 23
2023Learning How to Propagate Messages in Graph Neural Networkscited by 23
2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
2022Hypernetworks for Zero-shot Transfer in Reinforcement Learningcited by 18
2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 16
2023Behavior Alignment via Reward Function Optimizationcited by 16
2024In-context Exploration-Exploitation for Reinforcement Learningcited by 16
2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
2025SCAR: Shapley Credit Assignment for More Efficient RLHFcited by 13
2025LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generationcited by 13
2021A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learningcited by 11
2022Meta-Gradients in Non-Stationary Environmentscited by 11
2022LECO: Learnable Episodic Count for Task-Specific Intrinsic Rewardcited by 11
2023Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learningcited by 11
2024Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexitycited by 10
2021Learning to Shape Rewards using a Game of Two Partnerscited by 9
2021Toward a `Standard Model' of Machine Learningcited by 8
2021Continuous Control With Ensemble Deep Deterministic Policy Gradientscited by 8
2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 8
2022Learning Synthetic Environments and Reward Networks for Reinforcement Learningcited by 6
2024Automatic Environment Shaping is the Next Frontier in RLcited by 6
2021Model-Free Learning of Safe yet Effective Controllerscited by 5
2021Towards Practical Credit Assignment for Deep Reinforcement Learningcited by 5
2021Wasserstein Distance Maximizing Intrinsic Controlcited by 5
2021One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learningcited by 5
2022Evolutionary Action Selection for Gradient-based Policy Learningcited by 5
2022Open-Ended Reinforcement Learning with Neural Reward Functionscited by 5
2024Combining Automated Optimisation of Hyperparameters and Reward Shapecited by 5
2025Unsupervised Skill Discovery as Exploration for Learning Agile Locomotioncited by 5
2021Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspectivecited by 4
2022Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?cited by 4

50 most cited of 100 All 100 (JSON)