mlsanityData to 14 Oct 2026

Open-vocabulary Object Detection via Vision and Language Knowledge Distillation

arXiv 2104.13921 ↗First posted 28 Apr 2021cs.CV cs.AI cs.LGcited by 906

On the map: Knowledge distillation → Detection and dense prediction distillation

Abstract (arXiv)

We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask APr with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 APr. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP50 on PASCAL VOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the-art by 4.8 on novel AP and 11.4 on overall AP. Code and demo are open-sourced at https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.

Key facts. Cited by 906 papers in the map (data through 14 Oct 2026). Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation. Builds on 35 of its 55 references that are in the map. First posted on arXiv 28 Apr 2021.

Builds on

35 of 55 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2017Mask R-CNNcited by 7,277
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
23 more
  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  2. 2016YOLO9000: Better, Faster, Strongercited by 1,431
  3. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  4. 2018Deformable ConvNets v2: More Deformable, Better Resultscited by 670
  5. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  6. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  7. 2021Towards Open World Object Detectioncited by 297
  8. 2018Zero-Shot Object Detectioncited by 189
  9. 2015Weakly Supervised Deep Detection Networkscited by 183
  10. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  11. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  12. 2021Simple multi-dataset detectioncited by 127
  13. 2017Open Vocabulary Scene Parsingcited by 76
  14. 2020Object Detection with a Unified Label Space from Multiple Datasetscited by 65
  15. 2019Dont Even Look Once: Synthesizing Features for Zero-Shot Detectioncited by 58
  16. 2018Zero-Shot Object Detection: Learning to Simultaneously Recognize and Localize Novel Conceptscited by 48
  17. 2017Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precisioncited by 46
  18. 2014Zero Shot Recognition with Unreliable Attributescited by 39
  19. 2018Zero-Shot Object Detection by Hybrid Region Embeddingcited by 32
  20. 2020Synthesizing the Unseen for Zero-shot Object Detectioncited by 32
  21. 2019Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detectioncited by 31
  22. 2020Background Learnable Cascade for Zero-Shot Object Detectioncited by 31
  23. 2016Multi-Cue Zero-Shot Learning with Strong Supervisioncited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 106; 2023: 270; 2024: 241; 2025: 174; 2026: 109.

906 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  6. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
  7. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  9. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  10. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  11. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  12. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
38 more
  1. 2021Extract Free Dense Labels from CLIPcited by 583
  2. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  3. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  4. 2023LERF: Language Embedded Radiance Fieldscited by 543
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  7. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  8. 2022Multimodal Learning with Transformers: A Surveycited by 400
  9. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 375
  10. 2022Visual Language Maps for Robot Navigationcited by 365
  11. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 363
  12. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  13. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 315
  14. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  15. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 304
  16. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 300
  17. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  18. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  19. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  20. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 289
  21. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 280
  22. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  23. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  24. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  25. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 268
  26. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  27. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  28. 2023A Cookbook of Self-Supervised Learningcited by 250
  29. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  30. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  31. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  32. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  33. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  34. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  35. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  36. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 198
  37. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  38. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187

50 most cited of 906 All 906 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Sampling Bag of Views for Open-Vocabulary Object Detection
  2. 2021Open Vocabulary Object Detection with Pseudo Bounding-Box Labels
  3. 2024Unlocking Textual and Visual Wisdom: Open-Vocabulary 3D Object Detection Enhanced by Comprehensive Guidance from Text and Image
  4. 2023SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector
  5. 2025LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation
  6. 2025Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
  7. 2025Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting
  8. 2022Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations