# SPIN: Simultaneous Perception, Interaction and Navigation

arXiv: https://arxiv.org/abs/2405.07991 · HTML: https://mlsanity.com/paper/2405.07991/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://mlsanity.com/paper/2201.08117/) (2022)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Legged Locomotion in Challenging Terrains using Egocentric Vision](https://mlsanity.com/paper/2211.07638/) (2022)
- [Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion](https://mlsanity.com/paper/2210.10044/) (2022)
- [Learning Exploration Policies for Navigation](https://mlsanity.com/paper/1903.01959/) (2019)
- [In Defense of Classical Image Processing: Fast Depth Completion on the CPU](https://mlsanity.com/paper/1802.00036/) (2018)
- [DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments](https://mlsanity.com/paper/1809.08379/) (2018)
- [GOAT: GO to Any Thing](https://mlsanity.com/paper/2311.06430/) (2023)
- [OpenVSLAM: A Versatile Visual SLAM Framework](https://mlsanity.com/paper/1910.01122/) (2019)
- [RTAB-Map as an Open-Source Lidar and Visual SLAM Library for Large-Scale and Long-Term Online Operation](https://mlsanity.com/paper/2403.06341/) (2024)
- [Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation](https://mlsanity.com/paper/2112.05251/) (2021)
- [RobustNav: Towards Benchmarking Robustness in Embodied Navigation](https://mlsanity.com/paper/2106.04531/) (2021)
- [Stubborn: A Strong Baseline for Indoor Object Navigation](https://mlsanity.com/paper/2203.07359/) (2022)
- [Causal Policy Gradient for Whole-Body Mobile Manipulation](https://mlsanity.com/paper/2305.04866/) (2023)
- [A Holistic Approach to Reactive Mobile Manipulation](https://mlsanity.com/paper/2109.04749/) (2021)
- [Bayesian Imitation Learning for End-to-End Mobile Manipulation](https://mlsanity.com/paper/2202.07600/) (2022)
- [Enabling Failure Recovery for On-The-Move Mobile Manipulation](https://mlsanity.com/paper/2305.08351/) (2023)

## Built on by (most cited first, 29 of 29)

- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learning](https://mlsanity.com/paper/2407.15815/) (2024)
- [MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation](https://mlsanity.com/paper/2503.13446/) (2025)
- [DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation](https://mlsanity.com/paper/2411.04999/) (2024)
- [BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activities](https://mlsanity.com/paper/2503.05652/) (2025)
- [Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback](https://mlsanity.com/paper/2410.05434/) (2024)
- [AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation](https://mlsanity.com/paper/2507.01961/) (2025)
- [Local Policies Enable Zero-shot Long-horizon Manipulation](https://mlsanity.com/paper/2410.22332/) (2024)
- [SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation](https://mlsanity.com/paper/2602.16863/) (2026)
- [UMI-on-Air: Embodiment-Aware Guidance for Embodiment-Agnostic Visuomotor Policies](https://mlsanity.com/paper/2510.02614/) (2025)
- [Multi-Modal Grounded Planning and Efficient Replanning For Learning Embodied Agents with A Few Examples](https://mlsanity.com/paper/2412.17288/) (2024)
- [Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2409.14891/) (2024)
- [Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach](https://mlsanity.com/paper/2503.08306/) (2025)
- [Learning Panorama-Aware VLA for Mobile Manipulation with Whole-Body Teleoperation](https://mlsanity.com/paper/2608.02257/) (2026)
- [StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception](https://mlsanity.com/paper/2605.09989/) (2026)
- [Apple: Toward General Active Perception via Reinforcement Learning](https://mlsanity.com/paper/2505.06182/) (2025)
- [Distilling Realizable Students from Unrealizable Teachers](https://mlsanity.com/paper/2505.09546/) (2025)
- [What does really matter in image goal navigation?](https://mlsanity.com/paper/2507.01667/) (2025)
- [Kinaema: a recurrent sequence model for memory and pose in motion](https://mlsanity.com/paper/2510.20261/) (2025)
- [Mobile UMI: Cross-View Diffusion Policy with Decoupled Kinematics for Mobile Manipulation](https://mlsanity.com/paper/2605.20894/) (2026)
- [ProMi: An Efficient Prototype-Mixture Baseline for Few-Shot Segmentation with Bounding-Box Annotations](https://mlsanity.com/paper/2505.12547/) (2025)
- [Flexible Locomotion Learning with Diffusion Model Predictive Control](https://mlsanity.com/paper/2510.04234/) (2025)
- [Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2510.06277/) (2025)
- [MoPA: Coordinated Mobile Manipulation via Subsystem-Specific Perception Alignment](https://mlsanity.com/paper/2609.12081/) (2026)
- [CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigation](https://mlsanity.com/paper/2602.20055/) (2026)
- [Risk-Aware Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2603.04579/) (2026)
- [Whole-Body Mobile Manipulation using Offline Reinforcement Learning on Sub-optimal Controllers](https://mlsanity.com/paper/2604.12509/) (2026)
- [Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework](https://mlsanity.com/paper/2610.17089/) (2026)

## Abstract (arXiv)

While there has been remarkable progress recently in the fields of manipulation and locomotion, mobile manipulation remains a long-standing challenge. Compared to locomotion or static manipulation, a mobile system must make a diverse range of long-horizon tasks feasible in unstructured and dynamic environments. While the applications are broad and interesting, there are a plethora of challenges in developing these systems such as coordination between the base and arm, reliance on onboard perception for perceiving and interacting with the environment, and most importantly, simultaneously integrating all these parts together. Prior works approach the problem using disentangled modular skills for mobility and manipulation that are trivially tied together. This causes several limitations such as compounding errors, delays in decision-making, and no whole-body coordination. In this work, we present a reactive mobile manipulation framework that uses an active visual system to consciously perceive and react to its environment. Similar to how humans leverage whole-body and hand-eye coordination, we develop a mobile manipulator that exploits its ability to move and see, more specifically -- to move in order to see and to see in order to move. This allows it to not only move around and interact with its environment but also, choose "when" to perceive "what" using an active visual system. We observe that such an agent learns to navigate around complex cluttered scenarios while displaying agile whole-body coordination using only ego-vision without needing to create environment maps. Results visualizations and videos at https://spin-robot.github.io/