# PencilNet: Zero-Shot Sim-to-Real Transfer Learning for Robust Gate Perception in Autonomous Drone Racing

arXiv: https://arxiv.org/abs/2207.14131 · HTML: https://mlsanity.com/paper/2207.14131/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2022.

## Built on by (most cited first, 8 of 8)

- [YOLO-FEDER FusionNet: A Novel Deep Learning Architecture for Drone Detection](https://mlsanity.com/paper/2406.11641/) (2024)
- [A Map-free Deep Learning-based Framework for Gate-to-Gate Monocular Visual Navigation aboard Miniaturized Aerial Vehicles](https://mlsanity.com/paper/2503.05251/) (2025)
- [Performance Optimization of YOLO-FEDER FusionNet for Robust Drone Detection in Visually Complex Environments](https://mlsanity.com/paper/2509.14012/) (2025)
- [Continual Learning for Robust Gate Detection under Dynamic Lighting in Autonomous Drone Racing](https://mlsanity.com/paper/2405.01054/) (2024)
- [Towards Task-Oriented Flying: Framework, Infrastructure, and Principles](https://mlsanity.com/paper/2504.15129/) (2025)
- [AI-Hybrid TRNG: Kernel-Based Deep Learning for Near-Uniform Entropy Harvesting from Physical Noise](https://mlsanity.com/paper/2507.00145/) (2025)
- [Fuzzy Logic Theory-based Adaptive Reward Shaping for Robust Reinforcement Learning (FARS)](https://mlsanity.com/paper/2604.15772/) (2026)
- [Exp2VLA: Enabling Vision-Language-Action for Drone Navigation from Expert Demonstrations](https://mlsanity.com/paper/2607.03146/) (2026)

## Abstract (arXiv)

In autonomous and mobile robotics, one of the main challenges is the robust on-the-fly perception of the environment, which is often unknown and dynamic, like in autonomous drone racing. In this work, we propose a novel deep neural network-based perception method for racing gate detection -- PencilNet -- which relies on a lightweight neural network backbone on top of a pencil filter. This approach unifies predictions of the gates' 2D position, distance, and orientation in a single pose tuple. We show that our method is effective for zero-shot sim-to-real transfer learning that does not need any real-world training samples. Moreover, our framework is highly robust to illumination changes commonly seen under rapid flight compared to state-of-art methods. A thorough set of experiments demonstrates the effectiveness of this approach in multiple challenging scenarios, where the drone completes various tracks under different lighting conditions.