mlsanityData to 7 Oct 2026

Mind the Gap: Offline Policy Optimization for Imperfect Rewards

arXiv 2302.01667 ↗First posted 3 Feb 2023cs.LG cs.AIcited by 15

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewards are available, which inflicts substantial performance loss for RL agents. In this study, we propose a unified offline policy optimization approach, textitRGM (Reward Gap Minimization), which can smartly handle diverse types of imperfect rewards. RGM is formulated as a bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitation distribution matching w.r.t. some expert data; the lower layer solves a pessimistic RL problem with the corrected rewards. By exploiting the duality of the lower layer, we derive a tractable algorithm that enables sampled-based learning without any online interactions. Comprehensive experiments demonstrate that RGM achieves superior performance to existing methods under diverse settings of imperfect rewards. Further, RGM can effectively correct wrong or inconsistent rewards against expert preference and retrieve useful information from biased rewards.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 63 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

43 of 63 references.

Closest ancestors · 13

Same direction, newest first

  1. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  5. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning16 Sep
  8. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2021Convex optimizationcited by 2,479
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2018DeepMind Control Suitecited by 935
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  10. 2021Implicit Behavioral Cloningcited by 442
  11. 2021How to Train Your Energy-Based Modelscited by 271
  12. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
18 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2017Inverse Reward Designcited by 246
  3. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  4. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  5. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  6. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
  7. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 100
  8. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  9. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  10. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  11. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  12. 2021On the Expressivity of Markov Rewardcited by 71
  13. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  14. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  15. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  16. 2017A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agentscited by 18
  17. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  18. 2021Offline Inverse Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 7; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  2. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  4. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  5. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  6. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  7. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  8. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  9. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  10. 2024Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexitycited by 10
  11. 2024Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learningcited by 6
  12. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
3 more
  1. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  2. 2023A Fully Data-Driven Approach for Realistic Traffic Signal Control Using Offline Reinforcement Learningcited by 1
  3. 2026The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspectivecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  2. 2023Beyond Reward: Offline Preference-guided Policy Optimization
  3. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
  4. 2025Online Optimization for Offline Safe Reinforcement Learning
  5. 2025Offline Safe Policy Optimization From Heterogeneous Feedback
  6. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  7. 2026GOPO: Policy Optimization using Ranked Rewards
  8. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret