mlsanityData to 7 Oct 2026

Utilizing Prior Solutions for Reward Shaping and Composition in Entropy-Regularized Reinforcement Learning

arXiv 2212.01174 ↗First posted 2 Dec 2022cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning (RL), the ability to utilize prior knowledge from previously solved tasks can allow agents to quickly solve new problems. In some cases, these new problems may be approximately solved by composing the solutions of previously solved primitive tasks (task composition). Otherwise, prior knowledge can be used to adjust the reward function for a new problem, in a way that leaves the optimal policy unchanged but enables quicker learning (reward shaping). In this work, we develop a general framework for reward shaping and task composition in entropy-regularized RL. To do so, we derive an exact relation connecting the optimal soft value functions for two entropy-regularized RL problems with different reward functions and dynamics. We show how the derived relation leads to a general result for reward shaping in entropy-regularized RL. We then generalize this approach to derive an exact relation connecting optimal value functions for the composition of multiple tasks in entropy-regularized RL. We validate these theoretical contributions with experiments showing that reward shaping and task composition lead to faster learning in various settings.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 9 of its 18 references that are in the map. First posted on arXiv 2 Dec 2022.

Builds on

9 of 18 references.

References · 9

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016OpenAI Gymcited by 1,684
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  4. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  5. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  6. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  7. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  8. 2021Identifiability in inverse reinforcement learningcited by 55
  9. 2020A Boolean Task Algebra for Reinforcement Learningcited by 46

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 2.

8 papers.

  1. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 21
  2. 2025Average-Reward Soft Actor-Criticcited by 6
  3. 2024Boosting Soft Q-Learning by Boundingcited by 4
  4. 2023Bounding the Optimal Value Function in Compositional Reinforcement Learningcited by 3
  5. 2024Semi-supervised Anomaly Detection via Adaptive Reinforcement Learning-Enabled Method with Causal Inference for Sensor Signalscited by 2
  6. 2025Constrained Sliced Wasserstein Embeddingcited by 2
  7. 2023Leveraging Prior Knowledge in Reinforcement Learning via Double-Sided Bounds on the Value Functioncited by 0
  8. 2024Reinforcement Learning for Control of Non-Markovian Cellular Population Dynamicscited by 0