mlsanityData to 14 Oct 2026

Provably Efficient Reward Transfer in Reinforcement Learning with Discrete Markov Decision Processes

arXiv 2503.13414 ↗First posted 17 Mar 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we propose a new solution to reward adaptation (RA) in reinforcement learning, where the agent adapts to a target reward function based on one or more existing source behaviors learned a priori under the same domain dynamics but different reward functions. While learning the target behavior from scratch is possible, it is often inefficient given the available source behaviors. Our work introduces a new approach to RA through the manipulation of Q-functions. Assuming the target reward function is a known function of the source reward functions, we compute bounds on the Q-function and present an iterative process (akin to value iteration) to tighten these bounds. Such bounds enable action pruning in the target domain before learning even starts. We refer to this method as "Q-Manipulation" (Q-M). The iteration process assumes access to a lite-model, which is easy to provide or learn. We formally prove that Q-M, under discrete domains, does not affect the optimality of the returned policy and show that it is provably efficient in terms of sample complexity in a probabilistic sense. Q-M is evaluated in a variety of synthetic and simulation domains to demonstrate its effectiveness, generalizability, and practicality.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 31 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

13 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Boosting Soft Q-Learning by Bounding26 Jun

Other references · 12

  1. 2017Safe Reinforcement Learning via Shieldingcited by 449
  2. 2016The Option-Critic Architecturecited by 448
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  4. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  5. 2020Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decompositioncited by 127
  6. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 96
  7. 2019Provably Efficient Q-Learning with Low Switching Costcited by 78
  8. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 58
  9. 2024Sharing Knowledge in Multi-Task Deep Reinforcement Learningcited by 44
  10. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  11. 2020Sequential Transfer in Reinforcement Learning with a Generative Modelcited by 10
  12. 2023Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalitiescited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values
  2. 2025Model-Based Reinforcement Learning in Discrete-Action Non-Markovian Reward Decision Processes
  3. 2026Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach
  4. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching
  5. 2026Commit to the Bit: Reactive Reinforcement Learning Done Right
  6. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks
  7. 2026Adapting the Behavior of Reinforcement Learning Agents to Changing Action Spaces and Reward Functions
  8. 2026From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning