# Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop

arXiv: https://arxiv.org/abs/2506.10968 · HTML: https://mlsanity.com/paper/2506.10968/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 72 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [FAST: Efficient Action Tokenization for Vision-Language-Action Models](https://mlsanity.com/paper/2501.09747/) (2025)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [Flex Attention: A Programming Model for Generating Optimized Attention Kernels](https://mlsanity.com/paper/2412.05496/) (2024)
- [iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes](https://mlsanity.com/paper/2012.02924/) (2020)
- [Open-TeleVision: Teleoperation with Immersive Active Visual Feedback](https://mlsanity.com/paper/2407.01512/) (2024)
- [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021)
- [Revisiting Active Perception](https://mlsanity.com/paper/1603.02729/) (2016)
- [MuJoCo Playground](https://mlsanity.com/paper/2502.08844/) (2025)
- [Learning Visuotactile Skills with Two Multifingered Hands](https://mlsanity.com/paper/2404.16823/) (2024)

## Built on by (most cited first, 25 of 25)

- [Viser: Imperative, Web-based 3D Visualization in Python](https://mlsanity.com/paper/2507.22885/) (2025)
- [Thinking in 360deg: Humanoid Visual Search in the Wild](https://mlsanity.com/paper/2511.20351/) (2025)
- [PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era](https://mlsanity.com/paper/2509.12989/) (2025)
- [Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers](https://mlsanity.com/paper/2507.15833/) (2025)
- [ActiveMimic: Egocentric Video Pretraining with Active Perception](https://mlsanity.com/paper/2606.06194/) (2026)
- [Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategy](https://mlsanity.com/paper/2602.01939/) (2026)
- [Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generation](https://mlsanity.com/paper/2603.23491/) (2026)
- [DepthFocus: Controllable Depth Estimation for See-Through Scenes](https://mlsanity.com/paper/2511.16993/) (2025)
- [E3VS-Bench: A Benchmark for Viewpoint-Dependent Active Perception in 3D Gaussian Splatting Scenes](https://mlsanity.com/paper/2604.17969/) (2026)
- [TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning](https://mlsanity.com/paper/2605.07943/) (2026)
- [Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splatting](https://mlsanity.com/paper/2511.18140/) (2025)
- [FOVI: A biologically-inspired foveated interface for deep vision models](https://mlsanity.com/paper/2602.03766/) (2026)
- [LIME: Learning Intent-aware Camera Motion from Egocentric Video](https://mlsanity.com/paper/2607.02417/) (2026)
- [Gaze on the Prize: Shaping Visual Attention with Return-Guided Contrastive Learning](https://mlsanity.com/paper/2510.08442/) (2025)
- [Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reach](https://mlsanity.com/paper/2512.16456/) (2025)
- [Panoramic Affordance Prediction](https://mlsanity.com/paper/2603.15558/) (2026)
- [Policy-based Foveated Imaging and Perception](https://mlsanity.com/paper/2606.02565/) (2026)
- [Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation](https://mlsanity.com/paper/2606.23625/) (2026)
- [TRACER: Texture-Robust Affordance Chain-of-Thought for Deformable-Object Refinement](https://mlsanity.com/paper/2601.20208/) (2026)
- [Self-supervised pretraining for an iterative image size agnostic vision transformer](https://mlsanity.com/paper/2604.20392/) (2026)
- [LEAP: Learning Emergent Active Perception for Quadruped Navigation](https://mlsanity.com/paper/2609.17628/) (2026)
- [ActiveScale: Scaling Active Perception for Robots across Model, Data, and Hardware](https://mlsanity.com/paper/2609.18514/) (2026)
- [Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning](https://mlsanity.com/paper/2609.34550/) (2026)
- [EyeRobot 2.0: Active Gaze for Precise Manipulation without Wrist Cameras](https://mlsanity.com/paper/2610.03710/) (2026)
- [FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs](https://mlsanity.com/paper/2610.20064/) (2026)

## Abstract (arXiv)

Humans do not passively observe the visual world -- we actively look in order to act. Motivated by this principle, we introduce EyeRobot, a robotic system with gaze behavior that emerges from the need to complete real-world tasks. We develop a mechanical eyeball that can freely rotate to observe its surroundings and train a gaze policy to control it using reinforcement learning. We accomplish this by first collecting teleoperated demonstrations paired with a 360 camera. This data is imported into a simulation environment that supports rendering arbitrary eyeball viewpoints, allowing episode rollouts of eye gaze on top of robot demonstrations. We then introduce a BC-RL loop to train the hand and eye jointly: the hand (BC) agent is trained from rendered eye observations, and the eye (RL) agent is rewarded when the hand produces correct action predictions. In this way, hand-eye coordination emerges as the eye looks towards regions which allow the hand to complete the task. EyeRobot implements a foveal-inspired policy architecture allowing high resolution with a small compute budget, which we find also leads to the emergence of more stable fixation as well as improved ability to track objects and ignore distractors. We evaluate EyeRobot on five panoramic workspace manipulation tasks requiring manipulation in an arc surrounding the robot arm. Our experiments suggest EyeRobot exhibits hand-eye coordination behaviors which effectively facilitate manipulation over large workspaces with a single camera. See project site for videos: https://www.eyerobot.net/