# Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation

arXiv: https://arxiv.org/abs/2409.14891 · HTML: https://mlsanity.com/paper/2409.14891/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 27 of 27)

- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving](https://mlsanity.com/paper/2304.14365/) (2023)
- [Scene as Occupancy](https://mlsanity.com/paper/2306.02851/) (2023)
- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)
- [SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2108.06227/) (2021)
- [Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2105.14829/) (2021)
- [Momentum Contrastive Voxel-wise Representation Learning for Semi-supervised Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2105.07059/) (2021)
- [SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency](https://mlsanity.com/paper/2112.01001/) (2021)
- [GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction](https://mlsanity.com/paper/2402.16174/) (2024)
- [Stabilizing Off-Policy Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2207.00986/) (2022)
- [SPIN: Simultaneous Perception, Interaction and Navigation](https://mlsanity.com/paper/2405.07991/) (2024)
- [Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions](https://mlsanity.com/paper/1811.08067/) (2018)
- [SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering](https://mlsanity.com/paper/2210.15185/) (2022)
- [Closed-Loop Next-Best-View Planning for Target-Driven Grasping](https://mlsanity.com/paper/2207.10543/) (2022)
- [Affordance-Driven Next-Best-View Planning for Robotic Grasping](https://mlsanity.com/paper/2309.09556/) (2023)
- [Multi-View Picking: Next-best-view Reaching for Improved Grasping in Clutter](https://mlsanity.com/paper/1809.08564/) (2018)
- [Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perception](https://mlsanity.com/paper/2311.13793/) (2023)
- [ViSaRL: Visual Reinforcement Learning Guided by Human Saliency](https://mlsanity.com/paper/2403.10940/) (2024)
- [Active Perception and Representation for Robotic Manipulation](https://mlsanity.com/paper/2003.06734/) (2020)
- [Learning 6-DoF Grasping and Pick-Place Using Attention Focus](https://mlsanity.com/paper/1806.06134/) (2018)
- [Viewpoint Push Planning for Mapping of Unknown Confined Spaces](https://mlsanity.com/paper/2303.03126/) (2023)
- [Transferable Active Grasping and Real Embodied Dataset](https://mlsanity.com/paper/2004.13358/) (2020)
- [Gradient-based Local Next-best-view Planning for Improved Perception of Targeted Plant Nodes](https://mlsanity.com/paper/2311.16759/) (2023)

## Built on by (most cited first, 5 of 5)

- [FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correction](https://mlsanity.com/paper/2509.04018/) (2025)
- [BFA: Best-Feature-Aware Fusion for Multi-View Fine-grained Manipulation](https://mlsanity.com/paper/2502.11161/) (2025)
- [LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imagination](https://mlsanity.com/paper/2606.10862/) (2026)
- [Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splatting](https://mlsanity.com/paper/2511.18140/) (2025)
- [Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback](https://mlsanity.com/paper/2610.12549/) (2026)

## Abstract (arXiv)

In real-world scenarios, many robotic manipulation tasks are hindered by occlusions and limited fields of view, posing significant challenges for passive observation-based models that rely on fixed or wrist-mounted cameras. In this paper, we investigate the problem of robotic manipulation under limited visual observation and propose a task-driven asynchronous active vision-action model.Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach allows the agent to adjust a third-person camera to actively observe the environment based on the task goal, and subsequently infer the appropriate manipulation actions.We trained and evaluated our model on 8 viewpoint-constrained tasks in RLBench. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.