# Unifying Foundation Models with Quadrotor Control for Visual Tracking Beyond Object Categories

arXiv: https://arxiv.org/abs/2310.04781 · HTML: https://mlsanity.com/paper/2310.04781/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2023.

## Built on by (most cited first, 9 of 9)

- [Real-Time Out-of-Distribution Failure Prevention via Multi-Modal Reasoning](https://mlsanity.com/paper/2505.10547/) (2025)
- [STHN: Deep Homography Estimation for UAV Thermal Geo-localization with Satellite Imagery](https://mlsanity.com/paper/2405.20470/) (2024)
- [UASTHN: Uncertainty-Aware Deep Homography Estimation for UAV Satellite-Thermal Geo-localization](https://mlsanity.com/paper/2502.01035/) (2025)
- [Show and Grasp: Few-shot Semantic Segmentation for Robot Grasping through Zero-shot Foundation Models](https://mlsanity.com/paper/2404.12717/) (2024)
- [NOVA: Navigation via Object-Centric Visual Autonomy for High-Speed Target Tracking in Unstructured GPS-Denied Environments](https://mlsanity.com/paper/2506.18689/) (2025)
- [Spatial Language Likelihood Grounding Network for Bayesian Fusion of Human-Robot Observations](https://mlsanity.com/paper/2507.19947/) (2025)
- [HUNT: High-Speed UAV Navigation and Tracking in Unstructured Environments via Instantaneous Relative Frames](https://mlsanity.com/paper/2509.19452/) (2025)
- [Foundation models on the bridge: Semantic hazard detection and safety maneuvers for maritime autonomy with vision-language models](https://mlsanity.com/paper/2512.24470/) (2025)
- [FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking](https://mlsanity.com/paper/2606.29783/) (2026)

## Abstract (arXiv)

Visual control enables quadrotors to adaptively navigate using real-time sensory data, bridging perception with action. Yet, challenges persist, including generalization across scenarios, maintaining reliability, and ensuring real-time responsiveness. This paper introduces a perception framework grounded in foundation models for universal object detection and tracking, moving beyond specific training categories. Integral to our approach is a multi-layered tracker integrated with the foundation detector, ensuring continuous target visibility, even when faced with motion blur, abrupt light shifts, and occlusions. Complementing this, we introduce a model-free controller tailored for resilient quadrotor visual tracking. Our system operates efficiently on limited hardware, relying solely on an onboard camera and an inertial measurement unit. Through extensive validation in diverse challenging indoor and outdoor environments, we demonstrate our system's effectiveness and adaptability. In conclusion, our research represents a step forward in quadrotor visual tracking, moving from task-specific methods to more versatile and adaptable operations.