# Spatial Action Maps for Mobile Manipulation

arXiv: https://arxiv.org/abs/2004.09141 · HTML: https://mlsanity.com/paper/2004.09141/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2020.

## Built on by (most cited first, 30 of 31)

- [PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning](https://mlsanity.com/paper/2201.10029/) (2022)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [Waypoint-Based Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2307.14326/) (2023)
- [Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation](https://mlsanity.com/paper/2107.13545/) (2021)
- [Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations](https://mlsanity.com/paper/2408.04380/) (2024)
- [HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation](https://mlsanity.com/paper/2305.03942/) (2023)
- [Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control](https://mlsanity.com/paper/2103.00375/) (2021)
- [DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation](https://mlsanity.com/paper/2503.16806/) (2025)
- [Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusion](https://mlsanity.com/paper/2402.16075/) (2024)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [N2M2: Learning Navigation for Arbitrary Mobile Manipulation Motions in Unseen and Dynamic Environments](https://mlsanity.com/paper/2206.08737/) (2022)
- [Spatial Intention Maps for Multi-Agent Mobile Manipulation](https://mlsanity.com/paper/2103.12710/) (2021)
- [Multi-Task Learning with Sequence-Conditioned Transporter Networks](https://mlsanity.com/paper/2109.07578/) (2021)
- [Visual Perspective Taking for Opponent Behavior Modeling](https://mlsanity.com/paper/2105.05145/) (2021)
- [Towards Disturbance-Free Visual Mobile Manipulation](https://mlsanity.com/paper/2112.12612/) (2021)
- [DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning](https://mlsanity.com/paper/2402.05421/) (2024)
- [HACMan++: Spatially-Grounded Motion Primitives for Manipulation](https://mlsanity.com/paper/2407.08585/) (2024)
- [QDP: Learning to Sequentially Optimise Quasi-Static and Dynamic Manipulation Primitives for Robotic Cloth Manipulation](https://mlsanity.com/paper/2303.13320/) (2023)
- [LeTO: Learning Constrained Visuomotor Policy with Differentiable Trajectory Optimization](https://mlsanity.com/paper/2401.17500/) (2024)
- [Map It Anywhere (MIA): Empowering Bird's Eye View Mapping using Large-scale Public Data](https://mlsanity.com/paper/2407.08726/) (2024)
- [FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvement](https://mlsanity.com/paper/2607.01111/) (2026)
- [Learning Diffusion Policy from Primitive Skills for Robot Manipulation](https://mlsanity.com/paper/2601.01948/) (2026)
- [Learning Pneumatic Non-Prehensile Manipulation with a Mobile Blower](https://mlsanity.com/paper/2204.02390/) (2022)
- [ClothPPO: A Proximal Policy Optimization Enhancing Framework for Robotic Cloth Manipulation with Observation-Aligned Action Spaces](https://mlsanity.com/paper/2405.04549/) (2024)
- [IN-Sight: Interactive Navigation through Sight](https://mlsanity.com/paper/2408.00343/) (2024)
- [Brain-inspired Action Generation with Spiking Transformer Diffusion Policy Model](https://mlsanity.com/paper/2411.09953/) (2024)
- [PRISM: Performer RS-IMLE for Single-pass Multisensory Imitation Learning](https://mlsanity.com/paper/2602.02396/) (2026)
- [Visual Foresight With a Local Dynamics Model](https://mlsanity.com/paper/2206.14802/) (2022)
- [Push Smarter, Not Harder: Hierarchical RL-Diffusion Policy for Efficient Nonprehensile Manipulation](https://mlsanity.com/paper/2512.10099/) (2025)
- [The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning](https://mlsanity.com/paper/2608.26116/) (2026)

## Abstract (arXiv)

Typical end-to-end formulations for learning robotic navigation involve predicting a small set of steering command actions (e.g., step forward, turn left, turn right, etc.) from images of the current state (e.g., a bird's-eye view of a SLAM reconstruction). Instead, we show that it can be advantageous to learn with dense action representations defined in the same domain as the state. In this work, we present "spatial action maps," in which the set of possible actions is represented by a pixel map (aligned with the input image of the current state), where each pixel represents a local navigational endpoint at the corresponding scene location. Using ConvNets to infer spatial action maps from state images, action predictions are thereby spatially anchored on local visual features in the scene, enabling significantly faster learning of complex behaviors for mobile manipulation tasks with reinforcement learning. In our experiments, we task a robot with pushing objects to a goal location, and find that policies learned with spatial action maps achieve much better performance than traditional alternatives.