# Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splatting

arXiv: https://arxiv.org/abs/2511.18140 · HTML: https://mlsanity.com/paper/2511.18140/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 32 of its 38 references that are in the map.
- First posted on arXiv 22 Nov 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 32)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [RoMa: Robust Dense Feature Matching](https://mlsanity.com/paper/2305.15404/) (2023)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [Open-TeleVision: Teleoperation with Immersive Active Visual Feedback](https://mlsanity.com/paper/2407.01512/) (2024)
- [Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2306.17817/) (2023)
- [ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image](https://mlsanity.com/paper/2310.17994/) (2023)
- [InstantSplat: Sparse-view Gaussian Splatting in Seconds](https://mlsanity.com/paper/2403.20309/) (2024)
- [Uncertainty Guided Policy for Active Robotic 3D Reconstruction using Neural Radiance Fields](https://mlsanity.com/paper/2209.08409/) (2022)
- [Novel Demonstration Generation with Gaussian Splatting Enables Robust One-Shot Manipulation](https://mlsanity.com/paper/2504.13175/) (2025)
- [DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models](https://mlsanity.com/paper/2402.13181/) (2024)
- [Vision in Action: Learning Active Perception from Human Demonstrations](https://mlsanity.com/paper/2506.15666/) (2025)
- [View-Invariant Policy Learning via Zero-Shot Novel View Synthesis](https://mlsanity.com/paper/2409.03685/) (2024)
- [Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop](https://mlsanity.com/paper/2506.10968/) (2025)
- [Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation](https://mlsanity.com/paper/2409.17435/) (2024)
- [Closed-Loop Next-Best-View Planning for Target-Driven Grasping](https://mlsanity.com/paper/2207.10543/) (2022)
- [Affordance-Driven Next-Best-View Planning for Robotic Grasping](https://mlsanity.com/paper/2309.09556/) (2023)
- [RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation](https://mlsanity.com/paper/2310.03478/) (2023)
- [VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility](https://mlsanity.com/paper/2503.12609/) (2025)
- [Learning a Thousand Tasks in a Day](https://mlsanity.com/paper/2511.10110/) (2025)
- [Mobi-π: Mobilizing Your Robot Learning Policy](https://mlsanity.com/paper/2505.23692/) (2025)
- [One-Shot Dual-Arm Imitation Learning](https://mlsanity.com/paper/2503.06831/) (2025)

## Built on by (most cited first, 2 of 2)

- [Instant-Fold: In-Context Imitation Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2606.04269/) (2026)
- [LIME: Learning Intent-aware Camera Motion from Egocentric Video](https://mlsanity.com/paper/2607.02417/) (2026)

## Abstract (arXiv)

We propose Observer Actor (ObAct), a novel framework for active vision imitation learning in which the observer moves to optimal visual observations for the actor. We study ObAct on a dual-arm robotic system equipped with wrist-mounted cameras. At test time, ObAct dynamically assigns observer and actor roles: the observer arm constructs a 3D Gaussian Splatting (3DGS) representation from three images, virtually explores this to find an optimal camera pose, then moves to this pose; the actor arm then executes a policy using the observer's observations. This formulation enhances the clarity and visibility of both the object and the gripper in the policy's observations. As a result, we enable the training of ambidextrous policies on observations that remain closer to the occlusion-free training distribution, leading to more robust policies. We study this formulation with two existing imitation learning methods -- trajectory transfer and behavior cloning -- and experiments show that ObAct significantly outperforms static-camera setups: trajectory transfer improves by 145% without occlusion and 233% with occlusion, while behavior cloning improves by 75% and 143%, respectively. Videos are available at https://obact.github.io.