# Aggressive Perception-Aware Navigation using Deep Optical Flow Dynamics and PixelMPC

arXiv: https://arxiv.org/abs/2001.02307 · HTML: https://mlsanity.com/paper/2001.02307/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jan 2020.

## Built on by (most cited first, 6 of 6)

- [DATT: Deep Adaptive Trajectory Tracking for Quadrotor Control](https://mlsanity.com/paper/2310.09053/) (2023)
- [Pixel Motion as Universal Representation for Robot Control](https://mlsanity.com/paper/2505.07817/) (2025)
- [Deep Model Predictive Optimization](https://mlsanity.com/paper/2310.04590/) (2023)
- [Approximate Inverse Reinforcement Learning from Vision-based Imitation Learning](https://mlsanity.com/paper/2004.08051/) (2020)
- [Camera Motion Estimation from RGB-D-Inertial Scene Flow](https://mlsanity.com/paper/2404.17251/) (2024)
- [Approaches to Analysis and Design of AI-Based Autonomous Vehicles](https://mlsanity.com/paper/2509.12169/) (2025)

## Abstract (arXiv)

Recently, vision-based control has gained traction by leveraging the power of machine learning. In this work, we couple a model predictive control (MPC) framework to a visual pipeline. We introduce deep optical flow (DOF) dynamics, which is a combination of optical flow and robot dynamics. Using the DOF dynamics, MPC explicitly incorporates the predicted movement of relevant pixels into the planned trajectory of a robot. Our implementation of DOF is memory-efficient, data-efficient, and computationally cheap so that it can be computed in real-time for use in an MPC framework. The suggested Pixel Model Predictive Control (PixelMPC) algorithm controls the robot to accomplish a high-speed racing task while maintaining visibility of the important features (gates). This improves the reliability of vision-based estimators for localization and can eventually lead to safe autonomous flight. The proposed algorithm is tested in a photorealistic simulation with a high-speed drone racing task.