# VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots

arXiv: https://arxiv.org/abs/2503.22876 · HTML: https://mlsanity.com/paper/2503.22876/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 28 Mar 2025.

## Builds on (most cited first, 17 of 17)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Robot Operating System 2: Design, Architecture, and Uses In The Wild](https://mlsanity.com/paper/2211.07752/) (2022)
- [Deep Patch Visual Odometry](https://mlsanity.com/paper/2208.04726/) (2022)
- [Flightmare: A Flexible Quadrotor Simulator](https://mlsanity.com/paper/2009.00563/) (2020)
- [AlphaPilot: Autonomous Drone Racing](https://mlsanity.com/paper/2005.12813/) (2020)
- [Agilicious: Open-Source and Open-Hardware Agile Quadrotor for Vision-Based Flight](https://mlsanity.com/paper/2307.06100/) (2023)
- [Learning Perception-Aware Agile Flight in Cluttered Environments](https://mlsanity.com/paper/2210.01841/) (2022)
- [FlightGoggles: A Modular Framework for Photorealistic Camera, Exteroceptive Sensor, and Dynamics Simulation](https://mlsanity.com/paper/1905.11377/) (2019)
- [The Blackbird Dataset: A large-scale dataset for UAV perception in aggressive flight](https://mlsanity.com/paper/1810.01987/) (2018)
- [GapFlyt: Active Vision Based Minimalist Structure-less Gap Detection For Quadrotor Flight](https://mlsanity.com/paper/1802.05330/) (2018)
- [MorphEyes: Variable Baseline Stereo For Quadrotor Navigation](https://mlsanity.com/paper/2011.03077/) (2020)
- [Survey of Simulators for Aerial Robots: An Overview and In-Depth Systematic Comparisons](https://mlsanity.com/paper/2311.02296/) (2023)
- [U-Net for MAV-based Penstock Inspection: an Investigation of Focal Loss in Multi-class Segmentation for Corrosion Identification](https://mlsanity.com/paper/1809.06576/) (2018)
- [Visual Navigation for Autonomous Vehicles: An Open-source Hands-on Robotics Course at MIT](https://mlsanity.com/paper/2206.00777/) (2022)

## Abstract (arXiv)

Autonomous aerial robots are becoming commonplace in our lives. Hands-on aerial robotics courses are pivotal in training the next-generation workforce to meet the growing market demands. Such an efficient and compelling course depends on a reliable testbed. In this paper, we present VizFlyt, an open-source perception-centric Hardware-In-The-Loop (HITL) photorealistic testing framework for aerial robotics courses. We utilize pose from an external localization system to hallucinate real-time and photorealistic visual sensors using 3D Gaussian Splatting. This enables stress-free testing of autonomy algorithms on aerial robots without the risk of crashing into obstacles. We achieve over 100Hz of system update rate. Lastly, we build upon our past experiences of offering hands-on aerial robotics courses and propose a new open-source and open-hardware curriculum based on VizFlyt for the future. We test our framework on various course projects in real-world HITL experiments and present the results showing the efficacy of such a system and its large potential use cases. Code, datasets, hardware guides and demo videos are available at https://pear.wpi.edu/research/vizflyt.html