mlsanityData to 14 Oct 2026

Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation

arXiv 2409.14891 ↗First posted 23 Sep 2024cs.RO cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

In real-world scenarios, many robotic manipulation tasks are hindered by occlusions and limited fields of view, posing significant challenges for passive observation-based models that rely on fixed or wrist-mounted cameras. In this paper, we investigate the problem of robotic manipulation under limited visual observation and propose a task-driven asynchronous active vision-action model.Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach allows the agent to adjust a third-person camera to actively observe the environment based on the task goal, and subsequently infer the appropriate manipulation actions.We trained and evaluated our model on 8 viewpoint-constrained tasks in RLBench. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 27 of its 40 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

27 of 40 references.

References · 27

  1. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  2. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,476
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  4. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  5. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  6. 2023Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Drivingcited by 332
  7. 2023Scene as Occupancycited by 209
  8. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 122
  9. 2021SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentationcited by 86
  10. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  11. 2021Momentum Contrastive Voxel-wise Representation Learning for Semi-supervised Volumetric Medical Image Segmentationcited by 49
  12. 2021SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistencycited by 48
15 more
  1. 2024GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstructioncited by 39
  2. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  3. 2024SPIN: Simultaneous Perception, Interaction and Navigationcited by 29
  4. 2018Reinforcement Learning of Active Vision for Manipulating Objects under Occlusionscited by 25
  5. 2022SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Renderingcited by 22
  6. 2022Closed-Loop Next-Best-View Planning for Target-Driven Graspingcited by 19
  7. 2023Affordance-Driven Next-Best-View Planning for Robotic Graspingcited by 16
  8. 2018Multi-View Picking: Next-best-view Reaching for Improved Grasping in Cluttercited by 15
  9. 2023Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perceptioncited by 15
  10. 2024ViSaRL: Visual Reinforcement Learning Guided by Human Saliencycited by 12
  11. 2020Active Perception and Representation for Robotic Manipulationcited by 10
  12. 2018Learning 6-DoF Grasping and Pick-Place Using Attention Focuscited by 7
  13. 2023Viewpoint Push Planning for Mapping of Unknown Confined Spacescited by 5
  14. 2020Transferable Active Grasping and Real Embodied Datasetcited by 4
  15. 2023Gradient-based Local Next-best-view Planning for Improved Perception of Targeted Plant Nodescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correctioncited by 21
  2. 2025BFA: Best-Feature-Aware Fusion for Multi-View Fine-grained Manipulationcited by 10
  3. 2026LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imaginationcited by 3
  4. 2025Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splattingcited by 2
  5. 2026Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ActiveWAM: Evidence-Aware Active Vision for World-Action Models
  2. 2024Optimizing Active Perception for Learning Simultaneous Viewpoint Selection and Manipulation with Diffusion Policy
  3. 2026ActiveScale: Scaling Active Perception for Robots across Model, Data, and Hardware
  4. 2026ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation
  5. 2026Reflex: Enabling Fast and Predictive Vision-Language-Action Models for Reaction-Critical Manipulation
  6. 2026SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Robotics
  7. 2026ProWorld: Multi-View World Model with Proprioceptive Feedback for Closed-Loop Policy Evaluation
  8. 2026Move-Then-Operate: Behavioral Phasing for Human-Like Robotic Manipulation