# FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation

arXiv: https://arxiv.org/abs/1905.11377 · HTML: https://mlsanity.com/paper/1905.11377/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 22 of 22)

- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [Autonomous Drone Racing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.08624/) (2021)
- [OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control](https://mlsanity.com/paper/2309.12825/) (2023)
- [Learning Interactive Driving Policies via Data-driven Simulation](https://mlsanity.com/paper/2111.12137/) (2021)
- [SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum](https://mlsanity.com/paper/2412.16346/) (2024)
- [Variational Inference MPC using Tsallis Divergence](https://mlsanity.com/paper/2104.00241/) (2021)
- [Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents](https://mlsanity.com/paper/2201.02569/) (2022)
- [Multi-Fidelity Reinforcement Learning for Time-Optimal Quadrotor Re-planning](https://mlsanity.com/paper/2403.08152/) (2024)
- [RP-VIO: Robust Plane-based Visual-Inertial Odometry for Dynamic Environments](https://mlsanity.com/paper/2103.10400/) (2021)
- [Visual-based Safe Landing for UAVs in Populated Areas: Real-time Validation in Virtual Environments](https://mlsanity.com/paper/2203.13792/) (2022)
- [ABot-Earth 0.5: Generative 3D Earth Model](https://mlsanity.com/paper/2606.09967/) (2026)
- [Towards Task-Oriented Flying: Framework, Infrastructure, and Principles](https://mlsanity.com/paper/2504.15129/) (2025)
- [RCUKF: Data-Driven Modeling Meets Bayesian Estimation](https://mlsanity.com/paper/2508.04985/) (2025)
- [JuggleRL: Mastering Ball Juggling with a Quadrotor via Deep Reinforcement Learning](https://mlsanity.com/paper/2509.24892/) (2025)
- [Advancing AI Challenges for the United States Department of the Air Force](https://mlsanity.com/paper/2511.00267/) (2025)
- [HERCULES: An Open-Source Simulation Framework for Heterogeneous Multi-Robot SLAM, Collaborative Perception, and Exploration](https://mlsanity.com/paper/2606.22756/) (2026)
- [UAV-Borne Mapping Algorithms for Low-Altitude and High-Speed Drone Applications](https://mlsanity.com/paper/2401.06407/) (2024)
- [Cesium Tiles for High-realism Simulation and Comparing SLAM Results in Corresponding Virtual and Real-world Environments](https://mlsanity.com/paper/2401.07962/) (2024)
- [VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots](https://mlsanity.com/paper/2503.22876/) (2025)
- [Uncertainty Quantification Framework for Aerial and UAV Photogrammetry through Error Propagation](https://mlsanity.com/paper/2507.13486/) (2025)
- [SPIRAL: Self-Play Incremental Racing Algorithm for Learning in Multi-Drone Competitions](https://mlsanity.com/paper/2510.22568/) (2025)
- [Curriculum-Based Iterative Self-Play for Scalable Multi-Drone Racing](https://mlsanity.com/paper/2510.22570/) (2025)

## Abstract (arXiv)

FlightGoggles is a photorealistic sensor simulator for perception-driven robotic vehicles. The key contributions of FlightGoggles are twofold. First, FlightGoggles provides photorealistic exteroceptive sensor simulation using graphics assets generated with photogrammetry. Second, it provides the ability to combine (i) synthetic exteroceptive measurements generated in silico in real time and (ii) vehicle dynamics and proprioceptive measurements generated in motio by vehicle(s) in a motion-capture facility. FlightGoggles is capable of simulating a virtual-reality environment around autonomous vehicle(s). While a vehicle is in flight in the FlightGoggles virtual reality environment, exteroceptive sensors are rendered synthetically in real time while all complex extrinsic dynamics are generated organically through the natural interactions of the vehicle. The FlightGoggles framework allows for researchers to accelerate development by circumventing the need to estimate complex and hard-to-model interactions such as aerodynamics, motor mechanics, battery electrochemistry, and behavior of other agents. The ability to perform vehicle-in-the-loop experiments with photorealistic exteroceptive sensor simulation facilitates novel research directions involving, e.g., fast and agile autonomous flight in obstacle-rich environments, safe human interaction, and flexible sensor selection. FlightGoggles has been utilized as the main test for selecting nine teams that will advance in the AlphaPilot autonomous drone racing challenge. We survey approaches and results from the top AlphaPilot teams, which may be of independent interest.