# Taming Self-Training for Open-Vocabulary Object Detection

arXiv: https://arxiv.org/abs/2308.06412 · HTML: https://mlsanity.com/paper/2308.06412/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 58 of its 65 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 30 of 58)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Soft-NMS -- Improving Object Detection With One Line of Code](https://mlsanity.com/paper/1704.04503/) (2017)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://mlsanity.com/paper/2106.09018/) (2021)
- [Unbiased Teacher for Semi-Supervised Object Detection](https://mlsanity.com/paper/2102.09480/) (2021)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)

## Built on by (most cited first, 25 of 25)

- [DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2505.04410/) (2025)
- [Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction](https://mlsanity.com/paper/2412.00556/) (2024)
- [OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision](https://mlsanity.com/paper/2405.17913/) (2024)
- [MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection](https://mlsanity.com/paper/2407.21465/) (2024)
- [Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation](https://mlsanity.com/paper/2504.09480/) (2025)
- [Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception](https://mlsanity.com/paper/2508.11256/) (2025)
- [A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2503.10152/) (2025)
- [LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation](https://mlsanity.com/paper/2503.13794/) (2025)
- [Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2308.15846/) (2023)
- [SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detection](https://mlsanity.com/paper/2410.05650/) (2024)
- [C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detection](https://mlsanity.com/paper/2509.23316/) (2025)
- [NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2603.21069/) (2026)
- [MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2510.14792/) (2025)
- [Beyond Open Vocabulary: Multimodal Prompting for Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2602.01954/) (2026)
- [Robust Onion: Peeling Open Vocab Object Detectors Under Noise](https://mlsanity.com/paper/2606.26734/) (2026)
- [Modulating CNN Features with Pre-Trained ViT Representations for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2501.16981/) (2025)
- [State and Scene Enhanced Prototypes for Weakly Supervised Open-Vocabulary Object Detection](https://mlsanity.com/paper/2511.18012/) (2025)
- [Auto-Vocabulary 3D Object Detection](https://mlsanity.com/paper/2512.16077/) (2025)
- [GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection](https://mlsanity.com/paper/2603.27014/) (2026)
- [Generalization Under Scrutiny: Cross-Domain Detection Progresses, Pitfalls, and Persistent Challenges](https://mlsanity.com/paper/2604.08230/) (2026)
- [Beyond Visual Cues: Semantic-Driven Token Filtering and Expert Routing for Anytime Person ReID](https://mlsanity.com/paper/2604.15090/) (2026)
- [Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2604.23344/) (2026)
- [COVD: Continual Open-Vocabulary Object Detection with Novel Concept Injection](https://mlsanity.com/paper/2605.27116/) (2026)
- [LV-OSD: Language-Vision-Complementary Open-Set Object Detection](https://mlsanity.com/paper/2605.28271/) (2026)
- [ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification](https://mlsanity.com/paper/2606.09360/) (2026)

## Abstract (arXiv)

Recent studies have shown promising performance in open-vocabulary object detection (OVD) by utilizing pseudo labels (PLs) from pretrained vision and language models (VLMs). However, teacher-student self-training, a powerful and widely used paradigm to leverage PLs, is rarely explored for OVD. This work identifies two challenges of using self-training in OVD: noisy PLs from VLMs and frequent distribution changes of PLs. To address these challenges, we propose SAS-Det that tames self-training for OVD from two key perspectives. First, we present a split-and-fusion (SAF) head that splits a standard detection into an open-branch and a closed-branch. This design can reduce noisy supervision from pseudo boxes. Moreover, the two branches learn complementary knowledge from different training data, significantly enhancing performance when fused together. Second, in our view, unlike in closed-set tasks, the PL distributions in OVD are solely determined by the teacher model. We introduce a periodic update strategy to decrease the number of updates to the teacher, thereby decreasing the frequency of changes in PL distributions, which stabilizes the training process. Extensive experiments demonstrate SAS-Det is both efficient and effective. SAS-Det outperforms recent models of the same scale by a clear margin and achieves 37.4 AP50 and 29.1 APr on novel categories of the COCO and LVIS benchmarks, respectively. Code is available at urlhttps://github.com/xiaofeng94/SAS-Det.