Automatic Reward Shaping from Confounded Offline Data

arXiv 2505.11478 ↗First posted 16 May 2025cs.AI cs.LGcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where unobserved confounding cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 35 of its 75 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

35 of 75 references.

Closest ancestors · 1

Same direction, newest first

2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications22 Jul

Other references · 34

2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
2019Solving Rubik's Cube with a Robot Handcited by 734
2018Is Q-learning Provably Efficient?cited by 487
2023Efficient Online Reinforcement Learning with Offline Datacited by 363
2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 176

22 more

2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 106
2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 105
2018Confounding-Robust Policy Improvementcited by 76
2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 62
2020Q-learning with Logarithmic Regretcited by 60
2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 55
2021Fine-Grained Gap-Dependent Bounds for Tabular MDPs via Adaptive Multi-Step Bootstrapcited by 49
2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
2022Causal Imitation Learning with Unobserved Confounderscited by 41
2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
2019Reward Shaping via Meta-Learningcited by 32
2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 31
2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
2022Sequential Causal Imitation Learning with Unobserved Confounderscited by 25
2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
2023Automatic Intrinsic Reward Shaping for Exploration in Deep Reinforcement Learningcited by 12
2023Risk-Aware Reward Shaping of Reinforcement Learning Agents for Autonomous Drivingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 7
2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 2
2026Confounding Robust Continuous Control via Automatic Reward Shapingcited by 2
2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
2025Learning Upper Lower Value Envelopes to Shape Online RL: A Principled Approachcited by 0
2026From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RLcited by 0

Similar, not cited

Close in wording, no citation either way