mlsanityData to 14 Oct 2026

Confounding Robust Continuous Control via Automatic Reward Shaping

arXiv 2602.10305 ↗First posted 10 Feb 2026cs.LG cs.AI cs.ROcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reward shaping has been applied widely to accelerate Reinforcement Learning (RL) agents' training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control problems, remains largely under-explained. In this work, we propose to automatically learn a reward shaping function for continuous control problems from offline datasets, potentially contaminated by unobserved confounding variables. Specifically, our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values, which is then used as the potentials in the Potential-Based Reward Shaping (PBRS) framework. Our proposed reward shaping algorithm is tested with Soft-Actor-Critic (SAC) on multiple commonly used continuous control benchmarks and exhibits strong performance guarantees under unobserved confounders. More broadly, our work marks a solid first step towards confounding robust continuous control from a causal perspective. Code for training our reward shaping functions can be found at https://github.com/mateojuliani/confounding_robust_cont_control.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 22 of its 51 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

22 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Automatic Reward Shaping from Confounded Offline Data16 May
  2. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications22 Jul

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2019Solving Rubik's Cube with a Robot Handcited by 734
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  12. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
8 more
  1. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  2. 2018Confounding-Robust Policy Improvementcited by 76
  3. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  4. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  5. 2022Sequential Causal Imitation Learning with Unobserved Confounderscited by 25
  6. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  7. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  8. 2025PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Causal Graphs to Markovian States: Multi-Order State Exposure for Deep RLcited by 0
  2. 2026Scalable Causal Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning
  2. 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
  3. 2025Improving the Effectiveness of Potential-Based Reward Shaping in Reinforcement Learning
  4. 2025Shaping Sparse Rewards in Reinforcement Learning: A Semi-supervised Approach
  5. 2025Model-Based Reinforcement Learning Under Confounding
  6. 2026ARMS: Automatic Reward Shaping for Sparse-Reward Multi-Agent Reinforcement Learning
  7. 2024Adaptive Reward Design for Reinforcement Learning
  8. 2024ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization