mlsanityData to 7 Oct 2026

VLM-PL: Advanced Pseudo Labeling Approach for Class Incremental Object Detection via Vision-Language Model

arXiv 2403.05346 ↗First posted 8 Mar 2024cs.CVcited by 23

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

In the field of Class Incremental Object Detection (CIOD), creating models that can continuously learn like humans is a major challenge. Pseudo-labeling methods, although initially powerful, struggle with multi-scenario incremental learning due to their tendency to forget past knowledge. To overcome this, we introduce a new approach called Vision-Language Model assisted Pseudo-Labeling (VLM-PL). This technique uses Vision-Language Model (VLM) to verify the correctness of pseudo ground-truths (GTs) without requiring additional model training. VLM-PL starts by deriving pseudo GTs from a pre-trained detector. Then, we generate custom queries for each pseudo GT using carefully designed prompt templates that combine image and text features. This allows the VLM to classify the correctness through its responses. Furthermore, VLM-PL integrates refined pseudo and real GTs from upcoming training, effectively combining new and old knowledge. Extensive experiments conducted on the Pascal VOC and MS COCO datasets not only highlight VLM-PL's exceptional performance in multi-scenario but also illuminate its effectiveness in dual-scenario by achieving state-of-the-art results in both.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 95 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

76 of 95 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
38 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2018YOLOv3: An Incremental Improvementcited by 2,652
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Learning without Forgettingcited by 2,465
  9. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  12. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  13. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  14. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  15. 2017Continual Learning with Deep Generative Replaycited by 1,152
  16. 2021Grounded Language-Image Pre-trainingcited by 1,147
  17. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  18. 2016Modeling Context in Referring Expressionscited by 1,054
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  20. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  21. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  22. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  23. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  24. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  25. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  26. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  27. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  28. 2020Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detectioncited by 408
  29. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  30. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  31. 2019On Tiny Episodic Memories in Continual Learningcited by 299
  32. 2021Towards Open World Object Detectioncited by 297
  33. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  34. 2019Libra R-CNN: Towards Balanced Learning for Object Detectioncited by 255
  35. 2021TOOD: Task-aligned One-stage Object Detectioncited by 218
  36. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  37. 2019Class-incremental Learning via Deep Model Consolidationcited by 172
  38. 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 143

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 15; 2026: 5.

23 papers.

Sort:

By area:

  1. 2025CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answeringcited by 36
  2. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  3. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluationcited by 10
  4. 2024One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answeringcited by 6
  5. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  6. 2024Foundation Model-Powered 3D Few-Shot Class Incremental Learning via Training-free Adaptorcited by 3
  7. 2025Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuningcited by 3
  8. 2025No Labels, No Problem: Training Visual Reasoners with Multimodal Verifierscited by 3
  9. 2025Can Synthetic Images Conquer Forgetting? Beyond Unexplored Doubts in Few-Shot Class-Incremental Learningcited by 2
  10. 2025Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Modelscited by 2
  11. 2025DuET: Dual Incremental Object Detection via Exemplar-Free Task Arithmeticcited by 1
  12. 2025AutoVDC: Automated Vision Data Cleaning Using Vision-Language Modelscited by 1
11 more
  1. 2024CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detectioncited by 0
  2. 2025B-RIGHT: Benchmark Re-evaluation for Integrity in Generalized Human-Object Interaction Testingcited by 0
  3. 2025Fine-Tuning Florence2 for Enhanced Object Detection in Un-constructed Environments: Vision-Language Model Approachcited by 0
  4. 2025Exploiting Stability-Plasticity Asymmetry in Pretrained Detectors for Incremental Object Detectioncited by 0
  5. 2025Handling Imbalanced Pseudolabels for Vision-Language Models with Concept Alignment and Confusion-Aware Calibrated Margincited by 0
  6. 2025Parameterized Prompt for Incremental Object Detectioncited by 0
  7. 2026Generalization Under Scrutiny: Cross-Domain Detection Progresses, Pitfalls, and Persistent Challengescited by 0
  8. 2026HyLoVQA: Dynamic Hypernetwork-Generated Low-Rank Adaptation for Continual Visual Question Answeringcited by 0
  9. 2026MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuningcited by 0
  10. 2026Where Is the Bee? Detecting Tiny Pollinators with a Single Collaborative-Head Transformercited by 0
  11. 2026C2Path: Class-Conditional Pathway Decoupling for Vision-Language Incremental Object Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Object Detection with Pseudo Labels from VLMs using Per-Object Co-teaching
  2. 2024The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge
  3. 2025Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models
  4. 2025Auto-Labeling Data for Object Detection
  5. 2021VinVL: Revisiting Visual Representations in Vision-Language Models
  6. 2025MSPL: Multi-Step Pseudo-Labeling for Open-Vocabulary Object Detection
  7. 2025No Labels Needed: Zero-Shot Image Classification with Collaborative Self-Learning
  8. 2024MarvelOVD: Marrying Object Recognition and Vision-Language Models for Robust Open-Vocabulary Object Detection