# Vision Transformers for End-to-End Vision-Based Quadrotor Obstacle Avoidance

arXiv: https://arxiv.org/abs/2405.10391 · HTML: https://mlsanity.com/paper/2405.10391/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 16 May 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Masked World Models for Visual Control](https://mlsanity.com/paper/2206.14244/) (2022)
- [Learning High-Speed Flight in the Wild](https://mlsanity.com/paper/2110.05113/) (2021)
- [Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation](https://mlsanity.com/paper/2107.00644/) (2021)
- [Driving Policy Transfer via Modularity and Abstraction](https://mlsanity.com/paper/1804.09364/) (2018)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [Robust and Efficient Quadrotor Trajectory Generation for Fast Autonomous Flight](https://mlsanity.com/paper/1907.01531/) (2019)
- [Learning Perception-Aware Agile Flight in Cluttered Environments](https://mlsanity.com/paper/2210.01841/) (2022)
- [Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2204.04905/) (2022)
- [SEER: Safe Efficient Exploration for Aerial Robots using Learning to Predict Information Gain](https://mlsanity.com/paper/2209.11034/) (2022)
- [AvoidBench: A high-fidelity vision-based obstacle avoidance benchmarking suite for multi-rotors](https://mlsanity.com/paper/2301.07430/) (2023)
- [Vision Transformers For Weeds and Crops Classification Of High Resolution UAV Images](https://mlsanity.com/paper/2109.02716/) (2021)
- [Autonomous quadrotor obstacle avoidance based on dueling double deep recurrent Q-learning with monocular vision](https://mlsanity.com/paper/2002.03510/) (2020)
- [Improving Sample Efficiency of Value Based Models Using Attention and Vision Transformers](https://mlsanity.com/paper/2202.00710/) (2022)
- [Generating Large-Scale Trajectories Efficiently using Double Descriptions of Polynomials](https://mlsanity.com/paper/2011.02662/) (2020)

## Built on by (most cited first, 14 of 14)

- [SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum](https://mlsanity.com/paper/2412.16346/) (2024)
- [Spatially-Enhanced Recurrent Memory for Long-Range Mapless Navigation via End-to-End Reinforcement Learning](https://mlsanity.com/paper/2506.05997/) (2025)
- [Dashing for the Golden Snitch: Multi-Drone Time-Optimal Motion Planning with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.16720/) (2024)
- [RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation](https://mlsanity.com/paper/2502.02054/) (2025)
- [SIGN: Safety-Aware Image-Goal Navigation for Autonomous Drones via Reinforcement Learning](https://mlsanity.com/paper/2508.12394/) (2025)
- [Vision-Based Deep Reinforcement Learning of UAV Autonomous Navigation Using Privileged Information](https://mlsanity.com/paper/2412.06313/) (2024)
- [Fly360: Omnidirectional Obstacle Avoidance within Drone View](https://mlsanity.com/paper/2603.06573/) (2026)
- [Self-Supervised Learning to Fly using Efficient Semantic Segmentation and Metric Depth Estimation for Low-Cost Autonomous UAVs](https://mlsanity.com/paper/2510.16624/) (2025)
- [CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer](https://mlsanity.com/paper/2511.14111/) (2025)
- [PrivLLMSwarm: Privacy-Preserving LLM-Driven UAV Swarms for Secure IoT Surveillance](https://mlsanity.com/paper/2512.06747/) (2025)
- [An End-to-end Flight Control Network for High-speed UAV Obstacle Avoidance based on Event-Depth Fusion](https://mlsanity.com/paper/2603.27181/) (2026)
- [CMRL: Collision-Aware and Memory-Enhanced Reinforcement Learning for UAV Navigation in Multi-Scale Obstacle Environments](https://mlsanity.com/paper/2605.14810/) (2026)
- [Vision-Guided Outdoor Flight and Obstacle Evasion via Reinforcement Learning](https://mlsanity.com/paper/2605.24449/) (2026)
- [FPV-CRASH: A Dataset for Offline Crash Moment Localization in Drone First-Person-View Videos](https://mlsanity.com/paper/2610.19785/) (2026)

## Abstract (arXiv)

We demonstrate the capabilities of an attention-based end-to-end approach for high-speed vision-based quadrotor obstacle avoidance in dense, cluttered environments, with comparison to various state-of-the-art learning architectures. Quadrotor unmanned aerial vehicles (UAVs) have tremendous maneuverability when flown fast; however, as flight speed increases, traditional model-based approaches to navigation via independent perception, mapping, planning, and control modules breaks down due to increased sensor noise, compounding errors, and increased processing latency. Thus, learning-based, end-to-end vision-to-control networks have shown to have great potential for online control of these fast robots through cluttered environments. We train and compare convolutional, U-Net, and recurrent architectures against vision transformer (ViT) models for depth image-to-control in high-fidelity simulation, observing that ViT models are more effective than others as quadrotor speeds increase and in generalization to unseen environments, while the addition of recurrence further improves performance while reducing quadrotor energy cost across all tested flight speeds. We assess performance at speeds of up to 7m/s in simulation and hardware. To the best of our knowledge, this is the first work to utilize vision transformers for end-to-end vision-based quadrotor control.