mlsanityData to 14 Oct 2026

Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models

arXiv 2505.20612 ↗First posted 27 May 2025cs.CV cs.CL cs.LGcited by 20

On the map: Multimodal LLMs / vision-language models → Driving VLMs and open-vocabulary detection

Abstract (arXiv)

Vision-language models (VLMs) trained on internet-scale data achieve remarkable zero-shot detection performance on common objects like car, truck, and pedestrian. However, state-of-the-art models still struggle to generalize to out-of-distribution classes, tasks and imaging modalities not typically found in their pre-training. Rather than simply re-training VLMs on more visual data, we argue that one should align VLMs to new concepts with annotation instructions containing a few visual examples and rich textual descriptions. To this end, we introduce Roboflow100-VL, a large-scale collection of 100 multi-modal object detection datasets with diverse concepts not commonly found in VLM pre-training. We evaluate state-of-the-art models on our benchmark in zero-shot, few-shot, semi-supervised, and fully-supervised settings, allowing for comparison across data regimes. Notably, we find that VLMs like GroundingDINO and Qwen2.5-VL achieve less than 2% zero-shot accuracy on challenging medical imaging datasets within Roboflow100-VL, demonstrating the need for few-shot concept alignment. Lastly, we discuss our recent CVPR 2025 Foundational FSOD competition and share insights from the community. Notably, the winning team significantly outperforms our baseline by 17 mAP! Our code and dataset are available at https://github.com/roboflow/rf100-vl and https://universe.roboflow.com/rf100-vl/.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 107 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

52 of 107 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge18 Jun
  2. 2023Revisiting Few-Shot Object Detection with Vision-Language Models22 Dec
  3. 2023Scaling Open-Vocabulary Object Detection16 Jun
  4. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model28 Mar

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2025Qwen2.5-VL Technical Reportcited by 5,587
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
36 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  6. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  7. 2021Grounded Language-Image Pre-trainingcited by 1,156
  8. 2016Modeling Context in Referring Expressionscited by 1,063
  9. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  13. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  14. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  15. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  16. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  17. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  18. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  19. 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 425
  20. 2022Visual Classification via Description from Large Language Modelscited by 336
  21. 2020Frustratingly Simple Few-Shot Object Detectioncited by 282
  22. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  23. 2019Meta R-CNN : Towards General Solver for Instance-level Few-shot Learningcited by 182
  24. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 150
  25. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  26. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 71
  27. 2024The Neglected Tails in Vision-Language Modelscited by 63
  28. 2023Multi-modal Queried Object Detection in the Wildcited by 47
  29. 2024Better Call SAL: Towards Learning to Segment Anything in Lidarcited by 47
  30. 2019Growing a Brain: Fine-Tuning by Increasing Model Capacitycited by 43
  31. 2022Towards Long-Tailed 3D Detectioncited by 40
  32. 2024LW-DETR: A Transformer Replacement to YOLO for Real-Time Detectioncited by 36
  33. 2022Roboflow 100: A Rich, Multi-Domain Object Detection Benchmarkcited by 26
  34. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  35. 2025Towards Learning to Complete Anything in Lidarcited by 7
  36. 2023Thinking Like an Annotator: Generation of Dataset Labeling Instructionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 8.

20 papers.

Sort:

By area:

  1. 2025RF-DETR: Neural Architecture Search for Real-Time Detection Transformerscited by 75
  2. 2025Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysiscited by 38
  3. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  4. 2025Think Twice to See More: Iterative Visual Reasoning in Medical VLMscited by 13
  5. 2025SmokeBench: Evaluating Multimodal Large Language Models for Wildfire Smoke Detectioncited by 5
  6. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  7. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
  8. 2026Multi-label Instance-level Generalised Visual Grounding in Agriculturecited by 2
  9. 2026DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detectioncited by 2
  10. 2026A Closer Look at Cross-Domain Few-Shot Object Detection: Fine-Tuning Matters and Parallel Decoder Helpscited by 2
  11. 2025Container damage detection using advanced computer vision model Yolov12 vs Yolov11 vs RF-DETR A comparative analysiscited by 1
  12. 2025PixelArena: A benchmark for Pixel-Precision Visual Intelligencecited by 1
8 more
  1. 2026Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindnesscited by 1
  2. 2026SynthRender and I-AsSET: Open-Source Framework and Dataset for Bidirectional Sim-Real Transfer in Industrial Object Perceptioncited by 1
  3. 2026Your Model Already Knows Don't Teach It, Learn to Ask It: Soft Prompting for Few-Shot Adaptation of Vision-Language Modelscited by 1
  4. 2025ZERO: Industry-ready Vision Foundation Model with Multi-modal Promptscited by 0
  5. 2025In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopycited by 0
  6. 2025UniFlow: Zero-Shot LiDAR Scene Flow for Autonomous Vehiclescited by 0
  7. 2026Vision-Language Grounding as Bidirectional Concept Correspondencecited by 0
  8. 2026OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation
  2. 2025A Review of 3D Object Detection with Vision-Language Models
  3. 2021VinVL: Revisiting Visual Representations in Vision-Language Models
  4. 2025From Words to Wavelengths: VLMs for Few-Shot Multispectral Object Detection
  5. 2025ODOV: Benchmark the Open-Domain Open-Vocabulary Object Detection
  6. 2025Robust Object Detection with Pseudo Labels from VLMs using Per-Object Co-teaching
  7. 2025Auto-Labeling Data for Object Detection
  8. 2022OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network