# Reward Shaping via Meta-Learning

arXiv: https://arxiv.org/abs/1901.09330 · HTML: https://mlsanity.com/paper/1901.09330/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jan 2019.

## Built on by (most cited first, 30 of 32)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023)
- [Meta-learning PINN loss functions](https://mlsanity.com/paper/2107.05544/) (2021)
- [Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF](https://mlsanity.com/paper/2402.06886/) (2024)
- [Self-Supervised Online Reward Shaping in Sparse-Reward Environments](https://mlsanity.com/paper/2103.04529/) (2021)
- [Behavior Alignment via Reward Function Optimization](https://mlsanity.com/paper/2310.19007/) (2023)
- [Admissible Policy Teaching through Reward Design](https://mlsanity.com/paper/2201.02185/) (2022)
- [On The Sample Complexity Bounds In Bilevel Reinforcement Learning](https://mlsanity.com/paper/2503.17644/) (2025)
- [Learning to Shape Rewards using a Game of Two Partners](https://mlsanity.com/paper/2103.09159/) (2021)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Scenario-Assisted Deep Reinforcement Learning](https://mlsanity.com/paper/2202.04337/) (2022)
- [Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2405.15194/) (2024)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Learning Synthetic Environments and Reward Networks for Reinforcement Learning](https://mlsanity.com/paper/2202.02790/) (2022)
- [Automatic Reward Shaping from Confounded Offline Data](https://mlsanity.com/paper/2505.11478/) (2025)
- [BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping](https://mlsanity.com/paper/2409.05358/) (2024)
- [Reward prediction for representation learning and reward shaping](https://mlsanity.com/paper/2105.03172/) (2021)
- [On Assessing The Safety of Reinforcement Learning algorithms Using Formal Methods](https://mlsanity.com/paper/2111.04865/) (2021)
- [LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning](https://mlsanity.com/paper/2504.15472/) (2025)
- [Misalignment from Treating Means as Ends](https://mlsanity.com/paper/2507.10995/) (2025)
- [Model-Augmented Q-learning](https://mlsanity.com/paper/2102.03866/) (2021)
- [Black Box Meta-Learning Intrinsic Rewards](https://mlsanity.com/paper/2407.21546/) (2024)
- [Model Based Meta Learning of Critics for Policy Gradients](https://mlsanity.com/paper/2204.02210/) (2022)
- [Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards](https://mlsanity.com/paper/2207.14722/) (2022)
- [Visual processing in context of reinforcement learning](https://mlsanity.com/paper/2208.12525/) (2022)
- [Embedding Contextual Information through Reward Shaping in Multi-Agent Learning: A Case Study from Google Football](https://mlsanity.com/paper/2303.15471/) (2023)
- [Auxiliary Reward Generation with Transition Distance Representation Learning](https://mlsanity.com/paper/2402.07412/) (2024)
- [Context-aware Constrained Reinforcement Learning Based Energy-Efficient Power Scheduling for Non-stationary XR Data Traffic](https://mlsanity.com/paper/2503.09391/) (2025)
- [From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation](https://mlsanity.com/paper/2510.07624/) (2025)
- [Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approach](https://mlsanity.com/paper/2510.19528/) (2025)

## Abstract (arXiv)

Reward shaping is one of the most effective methods to tackle the crucial yet challenging problem of credit assignment in Reinforcement Learning (RL). However, designing shaping functions usually requires much expert knowledge and hand-engineering, and the difficulties are further exacerbated given multiple similar tasks to solve. In this paper, we consider reward shaping on a distribution of tasks, and propose a general meta-learning framework to automatically learn the efficient reward shaping on newly sampled tasks, assuming only shared state space but not necessarily action space. We first derive the theoretically optimal reward shaping in terms of credit assignment in model-free RL. We then propose a value-based meta-learning algorithm to extract an effective prior over the optimal reward shaping. The prior can be applied directly to new tasks, or provably adapted to the task-posterior while solving the task within few gradient updates. We demonstrate the effectiveness of our shaping through significantly improved learning efficiency and interpretable visualizations across various settings, including notably a successful transfer from DQN to DDPG.