# Toward Open Vocabulary Aerial Object Detection with CLIP-Activated Student-Teacher Learning

arXiv: https://arxiv.org/abs/2311.11646 · HTML: https://mlsanity.com/paper/2311.11646/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 74 references that are in the map.
- First posted on arXiv 20 Nov 2023.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [DOTA: A Large-scale Dataset for Object Detection in Aerial Images](https://mlsanity.com/paper/1711.10398/) (2017)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](https://mlsanity.com/paper/2306.11029/) (2023)
- [Object Detection in Optical Remote Sensing Images: A Survey and A New Benchmark](https://mlsanity.com/paper/1909.00133/) (2019)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://mlsanity.com/paper/2106.09018/) (2021)
- [GLIPv2: Unifying Localization and Vision-Language Understanding](https://mlsanity.com/paper/2206.05836/) (2022)
- [Detection and Tracking Meet Drones Challenge](https://mlsanity.com/paper/2001.06303/) (2020)

## Built on by (most cited first, 22 of 22)

- [Locate Anything on Earth: Advancing Open-Vocabulary Object Detection for Remote Sensing Community](https://mlsanity.com/paper/2408.09110/) (2024)
- [RemoteSAM: Towards Segment Anything for Earth Observation](https://mlsanity.com/paper/2505.18022/) (2025)
- [Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2509.21976/) (2025)
- [Securing the Skies: A Comprehensive Survey on Anti-UAV Methods, Benchmarking, and Future Directions](https://mlsanity.com/paper/2504.11967/) (2025)
- [OpenRSD: Towards Open-prompts for Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2503.06146/) (2025)
- [InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognition](https://mlsanity.com/paper/2505.15818/) (2025)
- [Exploiting Unlabeled Data with Multiple Expert Teachers for Open Vocabulary Aerial Object Detection and Its Orientation Adaptation](https://mlsanity.com/paper/2411.02057/) (2024)
- [Open-Vocabulary Object Detection in UAV Imagery: A Review and Future Perspectives](https://mlsanity.com/paper/2507.13359/) (2025)
- [OS-W2S: An Automatic Labeling Engine for Language-Guided Open-Set Aerial Object Detection](https://mlsanity.com/paper/2505.03334/) (2025)
- [Open-Text Aerial Detection: A Unified Framework For Aerial Visual Grounding And Detection](https://mlsanity.com/paper/2602.07827/) (2026)
- [GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning](https://mlsanity.com/paper/2603.09566/) (2026)
- [Cross-View Open-Vocabulary Object Detection in Aerial Imagery](https://mlsanity.com/paper/2510.03858/) (2025)
- [VK-Det: Visual Knowledge Guided Prototype Learning for Open-Vocabulary Aerial Object Detection](https://mlsanity.com/paper/2511.18075/) (2025)
- [Beyond Open Vocabulary: Multimodal Prompting for Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2602.01954/) (2026)
- [SLIP-RS: Structured-Attribute Language-Image Pre-Training for Remote Sensing Object Detection](https://mlsanity.com/paper/2605.23144/) (2026)
- [Enhancing Vehicle Detection under Adverse Weather Conditions with Contrastive Learning](https://mlsanity.com/paper/2509.21916/) (2025)
- [Auto-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2512.16077/) (2025)
- [Do Open-Vocabulary Detectors Transfer to Aerial Imagery? A Comparative Evaluation](https://mlsanity.com/paper/2601.22164/) (2026)
- [HMR-Net: Hierarchical Modular Routing for Cross-Domain Object Detection in Aerial Images](https://mlsanity.com/paper/2604.18866/) (2026)
- [DisDop: Distillation with Domain Priors for Open-Vocabulary Aerial Object Detection](https://mlsanity.com/paper/2605.24639/) (2026)
- [Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos](https://mlsanity.com/paper/2606.16124/) (2026)
- [Promptable Concept Segmentation from Above: Evaluating SAM 3's Zero-Shot and One-Shot Capabilities in Remote Sensing](https://mlsanity.com/paper/2607.09583/) (2026)

## Abstract (arXiv)

An increasingly massive number of remote-sensing images spurs the development of extensible object detectors that can detect objects beyond training categories without costly collecting new labeled data. In this paper, we aim to develop open-vocabulary object detection (OVD) technique in aerial images that scales up object vocabulary size beyond training data. The performance of OVD greatly relies on the quality of class-agnostic region proposals and pseudo-labels for novel object categories. To simultaneously generate high-quality proposals and pseudo-labels, we propose CastDet, a CLIP-activated student-teacher open-vocabulary object Detection framework. Our end-to-end framework following the student-teacher self-learning mechanism employs the RemoteCLIP model as an extra omniscient teacher with rich knowledge. By doing so, our approach boosts not only novel object proposals but also classification. Furthermore, we devise a dynamic label queue strategy to maintain high-quality pseudo labels during batch training. We conduct extensive experiments on multiple existing aerial object detection datasets, which are set up for the OVD task. Experimental results demonstrate our CastDet achieving superior open-vocabulary detection performance, e.g., reaching 46.5% mAP on VisDroneZSD novel categories, which outperforms the state-of-the-art open-vocabulary detectors by 21.0% mAP. To our best knowledge, this is the first work to apply and develop the open-vocabulary object detection technique for aerial images. The code is available at https://github.com/lizzy8587/CastDet.