# Causality-Based Reinforcement Learning Method for Multi-Stage Robotic Tasks

arXiv: https://arxiv.org/abs/2503.03145 · HTML: https://mlsanity.com/paper/2503.03145/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)
- [HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators](https://mlsanity.com/paper/1910.11432/) (2019)
- [A Survey on Causal Reinforcement Learning](https://mlsanity.com/paper/2302.05209/) (2023)
- [Causal Reinforcement Learning using Observational and Interventional Data](https://mlsanity.com/paper/2106.14421/) (2021)
- [Causality-driven Hierarchical Structure Discovery for Reinforcement Learning](https://mlsanity.com/paper/2210.06964/) (2022)
- [Causal Reasoning in Simulation for Structure and Transfer Learning of Robot Manipulation Policies](https://mlsanity.com/paper/2103.16772/) (2021)
- [Causal Policy Gradient for Whole-Body Mobile Manipulation](https://mlsanity.com/paper/2305.04866/) (2023)
- [Vision-based Navigation Using Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03627/) (2019)
- [Multi-Stage Reinforcement Learning for Non-Prehensile Manipulation](https://mlsanity.com/paper/2307.12074/) (2023)
- [Hierarchical Visual Policy Learning for Long-Horizon Robot Manipulation in Densely Cluttered Scenes](https://mlsanity.com/paper/2312.02697/) (2023)
- [Developing cooperative policies for multi-stage reinforcement learning tasks](https://mlsanity.com/paper/2205.05230/) (2022)
- [Learning Robotic Manipulation Tasks via Task Progress based Gaussian Reward and Loss Adjusted Exploration](https://mlsanity.com/paper/2103.01434/) (2021)
- [Learning Multi-step Robotic Manipulation Policies from Visual Observation of Scene and Q-value Predictions of Previous Action](https://mlsanity.com/paper/2202.11280/) (2022)

## Abstract (arXiv)

Deep reinforcement learning has made significant strides in various robotic tasks. However, employing deep reinforcement learning methods to tackle multi-stage tasks still a challenge. Reinforcement learning algorithms often encounter issues such as redundant exploration, getting stuck in dead ends, and progress reversal in multi-stage tasks. To address this, we propose a method that integrates causal relationships with reinforcement learning for multi-stage tasks. Our approach enables robots to automatically discover the causal relationships between their actions and the rewards of the tasks and constructs the action space using only causal actions, thereby reducing redundant exploration and progress reversal. By integrating correct causal relationships using the causal policy gradient method into the learning process, our approach can enhance the performance of reinforcement learning algorithms in multi-stage robotic tasks.