# Real-time Perception meets Reactive Motion Generation

arXiv: https://arxiv.org/abs/1703.03512 · HTML: https://mlsanity.com/paper/1703.03512/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2017.

## Built on by (most cited first, 30 of 30)

- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects](https://mlsanity.com/paper/2303.14158/) (2023)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration](https://mlsanity.com/paper/2201.12716/) (2022)
- [CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation](https://mlsanity.com/paper/2203.01929/) (2022)
- [Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks](https://mlsanity.com/paper/2405.01534/) (2024)
- [Local Policies Enable Zero-shot Long-horizon Manipulation](https://mlsanity.com/paper/2410.22332/) (2024)
- [Vision-driven Compliant Manipulation for Reliable, High-Precision Assembly Tasks](https://mlsanity.com/paper/2106.14070/) (2021)
- [Composable Energy Policies for Reactive Motion Generation and Reinforcement Learning](https://mlsanity.com/paper/2105.04962/) (2021)
- [Zero-Shot Multi-Object Scene Completion](https://mlsanity.com/paper/2403.14628/) (2024)
- [RMP2: A Structured Composable Policy Class for Robot Learning](https://mlsanity.com/paper/2103.05922/) (2021)
- [GeoReF: Geometric Alignment Across Shape Variation for Category-level Object Pose Refinement](https://mlsanity.com/paper/2404.11139/) (2024)
- [Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization](https://mlsanity.com/paper/2403.14973/) (2024)
- [Industry 6.0: New Generation of Industry driven by Generative AI and Swarm of Heterogeneous Robots](https://mlsanity.com/paper/2409.10106/) (2024)
- [A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewards](https://mlsanity.com/paper/2502.08643/) (2025)
- [Unified Category-Level Object Detection and Pose Estimation from RGB Images using 3D Prototypes](https://mlsanity.com/paper/2508.02157/) (2025)
- [Diffusion-Based Approximate MPC: Fast and Consistent Imitation of Multi-Modal Action Distributions](https://mlsanity.com/paper/2504.04603/) (2025)
- [MUSE: Model-based Uncertainty-aware Similarity Estimation for zero-shot 2D Object Detection and Segmentation](https://mlsanity.com/paper/2510.17866/) (2025)
- [Deep 6-DoF Tracking of Unknown Objects for Reactive Grasping](https://mlsanity.com/paper/2103.05401/) (2021)
- [TrackAgent: 6D Object Tracking via Reinforcement Learning](https://mlsanity.com/paper/2307.15671/) (2023)
- [ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization](https://mlsanity.com/paper/2401.08937/) (2024)
- [CoverLib: Classifiers-equipped Experience Library by Iterative Problem Distribution Coverage Maximization for Domain-tuned Motion Planning](https://mlsanity.com/paper/2405.02968/) (2024)
- [Joint Learning of Pose Regression and Denoising Diffusion with Score Scaling Sampling for Category-level 6D Pose Estimation](https://mlsanity.com/paper/2510.04125/) (2025)
- [SuperPose: Improved 6D Pose Estimation with Robust Tracking and Mask-Free Initialization](https://mlsanity.com/paper/2409.19986/) (2024)
- [MVTOP: Multi-View Transformer-based Object Pose-Estimation](https://mlsanity.com/paper/2508.03243/) (2025)
- [SAGE: Scene Graph-Aware Guidance and Execution for Long-Horizon Manipulation Tasks](https://mlsanity.com/paper/2509.21928/) (2025)
- [STORM: Segment, Track, and Object Re-Localization from a Single Image](https://mlsanity.com/paper/2511.09771/) (2025)
- [Yolo-Key-6D: Single Stage Monocular 6D Pose Estimation with Keypoint Enhancements](https://mlsanity.com/paper/2603.03879/) (2026)
- [SplatCtrl: Perception-Action Coupling via Gaussian Scene Representations and Reactive Robot Control](https://mlsanity.com/paper/2607.08948/) (2026)

## Abstract (arXiv)

We address the challenging problem of robotic grasping and manipulation in the presence of uncertainty. This uncertainty is due to noisy sensing, inaccurate models and hard-to-predict environment dynamics. We quantify the importance of continuous, real-time perception and its tight integration with reactive motion generation methods in dynamic manipulation scenarios. We compare three different systems that are instantiations of the most common architectures in the field: (i) a traditional sense-plan-act approach that is still widely used, (ii) a myopic controller that only reacts to local environment dynamics and (iii) a reactive planner that integrates feedback control and motion optimization. All architectures rely on the same components for real-time perception and reactive motion generation to allow a quantitative evaluation. We extensively evaluate the systems on a real robotic platform in four scenarios that exhibit either a challenging workspace geometry or a dynamic environment. In 333 experiments, we quantify the robustness and accuracy that is due to integrating real-time feedback at different time scales in a reactive motion generation system. We also report on the lessons learned for system building.