# Causal Policy Gradient for Whole-Body Mobile Manipulation

arXiv: https://arxiv.org/abs/2305.04866 · HTML: https://mlsanity.com/paper/2305.04866/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 62 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [Sim-to-Real Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/1806.07851/) (2018)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation](https://mlsanity.com/paper/1703.00420/) (2017)
- [Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines](https://mlsanity.com/paper/1803.07246/) (2018)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)
- [Data-efficient Deep Reinforcement Learning for Dexterous Manipulation](https://mlsanity.com/paper/1704.03073/) (2017)
- [HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators](https://mlsanity.com/paper/1910.11432/) (2019)
- [Causal Discovery from Changes](https://mlsanity.com/paper/1301.2312/) (2013)
- [ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2008.07792/) (2020)
- [The Mirage of Action-Dependent Baselines in Reinforcement Learning](https://mlsanity.com/paper/1802.10031/) (2018)
- [Learning Dexterous Manipulation for a Soft Robotic Hand from Human Demonstration](https://mlsanity.com/paper/1603.06348/) (2016)
- [Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning](https://mlsanity.com/paper/2010.08600/) (2020)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [A Holistic Approach to Reactive Mobile Manipulation](https://mlsanity.com/paper/2109.04749/) (2021)
- [N2M2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments](https://mlsanity.com/paper/2206.08737/) (2022)
- [Vision-based Navigation Using Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03627/) (2019)
- [Whole-Body Control of a Mobile Manipulator using End-to-End Reinforcement Learning](https://mlsanity.com/paper/2003.02637/) (2020)
- [Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPs](https://mlsanity.com/paper/2102.10362/) (2021)

## Built on by (most cited first, 21 of 21)

- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning](https://mlsanity.com/paper/2505.18719/) (2025)
- [FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning](https://mlsanity.com/paper/2409.16578/) (2024)
- [BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities](https://mlsanity.com/paper/2503.05652/) (2025)
- [SPIN: Simultaneous Perception, Interaction and Navigation](https://mlsanity.com/paper/2405.07991/) (2024)
- [M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.11402/) (2024)
- [Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI](https://mlsanity.com/paper/2310.01824/) (2023)
- [Harmonic Mobile Manipulation](https://mlsanity.com/paper/2312.06639/) (2023)
- [UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policies](https://mlsanity.com/paper/2510.02614/) (2025)
- [InCoRo: In-Context Learning for Robotics Control with Feedback Loops](https://mlsanity.com/paper/2402.05188/) (2024)
- [Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2410.11251/) (2024)
- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [ELDEN: Exploration via Local Dependencies](https://mlsanity.com/paper/2310.08702/) (2023)
- [SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Training](https://mlsanity.com/paper/2506.04147/) (2025)
- [MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment](https://mlsanity.com/paper/2312.03644/) (2023)
- [Learning Any-View 6DoF Robotic Grasping in Cluttered Scenes via Neural Surface Rendering](https://mlsanity.com/paper/2306.07392/) (2023)
- [Versatile Loco-Manipulation through Flexible Interlimb Coordination](https://mlsanity.com/paper/2506.07876/) (2025)
- [MoPA: Coordinated Mobile Manipulation via Subsystem-Specific Perception Alignment](https://mlsanity.com/paper/2609.12081/) (2026)
- [Causality-Based Reinforcement Learning Method for Multi-Stage Robotic Tasks](https://mlsanity.com/paper/2503.03145/) (2025)
- [Hierarchical Residual Policy Learning for Real-World Mobile Manipulation with Sparse Human Guidance](https://mlsanity.com/paper/2610.20607/) (2026)

## Abstract (arXiv)

Developing the next generation of household robot helpers requires combining locomotion and interaction capabilities, which is generally referred to as mobile manipulation (MoMa). MoMa tasks are difficult due to the large action space of the robot and the common multi-objective nature of the task, e.g., efficiently reaching a goal while avoiding obstacles. Current approaches often segregate tasks into navigation without manipulation and stationary manipulation without locomotion by manually matching parts of the action space to MoMa sub-objectives (e.g. learning base actions for locomotion objectives and learning arm actions for manipulation). This solution prevents simultaneous combinations of locomotion and interaction degrees of freedom and requires human domain knowledge for both partitioning the action space and matching the action parts to the sub-objectives. In this paper, we introduce Causal MoMa, a new reinforcement learning framework to train policies for typical MoMa tasks that makes use of the most favorable subspace of the robot's action space to address each sub-objective. Causal MoMa automatically discovers the causal dependencies between actions and terms of the reward function and exploits these dependencies through causal policy gradient that reduces gradient variance compared to previous state-of-the-art reinforcement learning algorithms, improving convergence and results. We evaluate the performance of Causal MoMa on three types of simulated robots across different MoMa tasks and demonstrate success in transferring the policies trained in simulation directly to a real robot, where our agent is able to follow moving goals and react to dynamic obstacles while simultaneously and synergistically controlling the whole-body: base, arm, and head. More information at https://sites.google.com/view/causal-moma.