# Seeing Through Pixel Motion: Learning Obstacle Avoidance from Optical Flow with One Camera

arXiv: https://arxiv.org/abs/2411.04413 · HTML: https://mlsanity.com/paper/2411.04413/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2024.

## Built on by (most cited first, 13 of 13)

- [Pixel Motion as Universal Representation for Robot Control](https://mlsanity.com/paper/2505.07817/) (2025)
- [GRaD-Nav++: Vision-Language Model Enabled Visual Drone Navigation with Gaussian Radiance Fields and Differentiable Dynamics](https://mlsanity.com/paper/2506.14009/) (2025)
- [Flying on Point Clouds with Reinforcement Learning](https://mlsanity.com/paper/2503.00496/) (2025)
- [SIGN: Safety-Aware Image-Goal Navigation for Autonomous Drones via Reinforcement Learning](https://mlsanity.com/paper/2508.12394/) (2025)
- [DiffAero: A GPU-Accelerated Differentiable Simulation Framework for Efficient Quadrotor Policy Learning](https://mlsanity.com/paper/2509.10247/) (2025)
- [Fly360: Omnidirectional Obstacle Avoidance within Drone View](https://mlsanity.com/paper/2603.06573/) (2026)
- [Curriculum Reinforcement Learning for Quadrotor Racing with Random Obstacles](https://mlsanity.com/paper/2602.24030/) (2026)
- [ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards](https://mlsanity.com/paper/2501.14513/) (2025)
- [Understanding visual attention beehind bee-inspired UAV navigation](https://mlsanity.com/paper/2507.11992/) (2025)
- [Zero-Shot UAV Navigation in Forests via Relightable 3D Gaussian Splatting](https://mlsanity.com/paper/2602.07101/) (2026)
- [GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields](https://mlsanity.com/paper/2604.05062/) (2026)
- [Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic](https://mlsanity.com/paper/2609.30696/) (2026)
- [UniDynamics: Event-RGB Fusion for Unified Future 4D Dynamic Scene Generation](https://mlsanity.com/paper/2610.03473/) (2026)

## Abstract (arXiv)

Optical flow captures the motion of pixels in an image sequence over time, providing information about movement, depth, and environmental structure. Flying insects utilize this information to navigate and avoid obstacles, allowing them to execute highly agile maneuvers even in complex environments. Despite its potential, autonomous flying robots have yet to fully leverage this motion information to achieve comparable levels of agility and robustness. Challenges of control from optical flow include extracting accurate optical flow at high speeds, handling noisy estimation, and ensuring robust performance in complex environments. To address these challenges, we propose a novel end-to-end system for quadrotor obstacle avoidance using monocular optical flow. We develop an efficient differentiable simulator coupled with a simplified quadrotor model, allowing our policy to be trained directly through first-order gradient optimization. Additionally, we introduce a central flow attention mechanism and an action-guided active sensing strategy that enhances the policy's focus on task-relevant optical flow observations to enable more responsive decision-making during flight. Our system is validated both in simulation and the real world using an FPV racing drone. Despite being trained in a simple environment in simulation, our system is validated both in simulation and the real world using an FPV racing drone. Despite being trained in a simple environment in simulation, our system demonstrates agile and robust flight in various unknown, cluttered environments in the real world at speeds of up to 6m/s.