# Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic

arXiv: https://arxiv.org/abs/2609.30696 · HTML: https://mlsanity.com/paper/2609.30696/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [Accelerated Policy Learning with Parallel Differentiable Simulation](https://mlsanity.com/paper/2204.07137/) (2022)
- [Back to Newton's Laws: Learning Vision-based Agile Flight via Differentiable Physics](https://mlsanity.com/paper/2407.10648/) (2024)
- [Learning Quadruped Locomotion Using Differentiable Simulation](https://mlsanity.com/paper/2403.14864/) (2024)
- [Demonstrating Agile Flight from Pixels without State Estimation](https://mlsanity.com/paper/2406.12505/) (2024)
- [Training Efficient Controllers via Analytic Policy Gradient](https://mlsanity.com/paper/2209.13052/) (2022)
- [Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning for Agile Flight](https://mlsanity.com/paper/2306.09852/) (2023)
- [Learning Quadrotor Control From Visual Features Using Differentiable Simulation](https://mlsanity.com/paper/2410.15979/) (2024)
- [Search-based Motion Planning for Aggressive Flight in SE(3)](https://mlsanity.com/paper/1710.02748/) (2017)
- [Whole-Body Control Through Narrow Gaps From Pixels To Action](https://mlsanity.com/paper/2409.00895/) (2024)
- [Seeing Through Pixel Motion: Learning Obstacle Avoidance from Optical Flow with One Camera](https://mlsanity.com/paper/2411.04413/) (2024)
- [Flying Through a Narrow Gap Using End-to-end Deep Reinforcement Learning Augmented with Curriculum Learning and Sim2Real](https://mlsanity.com/paper/2108.12869/) (2021)
- [Learning Agile Flights through Narrow Gaps with Varying Angles using Onboard Sensing](https://mlsanity.com/paper/2302.11233/) (2023)
- [Aggressive Quadrotor Flight through Narrow Gaps with Onboard Sensing and Computing using Active Vision](https://mlsanity.com/paper/1612.00291/) (2016)
- [OmniNxt: A Fully Open-source and Compact Aerial Robot with Omnidirectional Visual Perception](https://mlsanity.com/paper/2403.20085/) (2024)
- [Learning Agile Flight Maneuvers: Deep SE(3) Motion Planning and Control for Quadrotors](https://mlsanity.com/paper/2209.11097/) (2022)
- [Accelerating Visual-Policy Learning through Parallel Differentiable Simulation](https://mlsanity.com/paper/2505.10646/) (2025)
- [Flying through a narrow gap using neural network: an end-to-end planning and control approach](https://mlsanity.com/paper/1903.09088/) (2019)
- [Precise Aggressive Aerial Maneuvers with Sensorimotor Policies](https://mlsanity.com/paper/2604.05828/) (2026)
- [Vector Field Augmented Differentiable Policy Learning for Vision-Based Drone Racing](https://mlsanity.com/paper/2603.08019/) (2026)
- [Learning Agile Gate Traversal via Analytical Optimal Policy Gradient](https://mlsanity.com/paper/2508.21592/) (2025)
- [Vision-Based End-to-End Learning for UAV Traversal of Irregular Gaps via Differentiable Simulation](https://mlsanity.com/paper/2604.02779/) (2026)

## Abstract (arXiv)

Traversing narrow gaps is challenging for autonomous quadrotors, especially when control commands come directly from high-dimensional visual observations. Existing end-to-end methods often rely on behavior cloning or full-rollout backpropagation through time (BPTT) via differentiable simulation, which can limit policy performance or incur high training costs. We propose a two-stage reinforcement learning framework for more efficient ego-centric visuomotor gap-traversal policy training, leveraging quasi-analytical policy gradients (QPG) via differentiable simulation and critic warm-starting. The framework utilizes QPG to avoid backpropagation through visual rendering, reducing computation and memory costs while improving sample efficiency. In the first stage, an expert actor and critic are trained using privileged observations, including gap geometry. Unlike prior gap-traversal approaches, our training utilizing QPG does not require resetting the agent along optimized reference trajectories. In the second stage, a visual policy is trained using binary gap masks from two ego-centric cameras and low-dimensional observations, while its privileged critic is warm-started from the first stage. This substantially improves training efficiency and traversal success compared with cold-starting the critic or using full-rollout BPTT. Our framework does not require retraining the expert actor when system parameters change, enabling more efficient generalization across drone platforms than state-of-the-art visual gap-traversal methods based on action supervision. The learned visual policy also generalizes to gaps with unseen shapes. Extensive real-world experiments further demonstrate robust gap traversal using binary masks rendered online. Beyond gap traversal, the proposed framework is generic and can be extended to other visuomotor robot learning tasks.