mlsanityData to 7 Oct 2026

Described Object Detection: Liberating Object Detection with Flexible Expressions

arXiv 2307.12813 ↗First posted 24 Jul 2023cs.CVcited by 40

Not on the map. Why?

Abstract (arXiv)

Detecting objects based on language information is a popular task that includes Open-Vocabulary object Detection (OVD) and Referring Expression Comprehension (REC). In this paper, we advance them to a more practical setting called Described Object Detection (DOD) by expanding category names to flexible language expressions for OVD and overcoming the limitation of REC only grounding the pre-existing object. We establish the research foundation for DOD by constructing a Description Detection Dataset (D3). This dataset features flexible language expressions, whether short category names or long descriptions, and annotating all described objects on all images without omission. By evaluating previous SOTA methods on D3, we find some troublemakers that fail current REC, OVD, and bi-functional methods. REC methods struggle with confidence scores, rejecting negative instances, and multi-target scenarios, while OVD methods face constraints with long and complex descriptions. Recent bi-functional methods also do not work well on DOD due to their separated training procedures and inference strategies for REC and OVD tasks. Building upon the aforementioned findings, we propose a baseline that largely improves REC methods by reconstructing the training data and introducing a binary classification sub-task, outperforming existing methods. Data and code are available at https://github.com/shikras/d-cube and related works are tracked in https://github.com/Charles-Xie/awesome-described-object-detection.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 49 of its 62 references that are in the map. First posted on arXiv 24 Jul 2023.

Builds on

49 of 62 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
37 more
  1. 2016Modeling Context in Referring Expressionscited by 1,054
  2. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  3. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  5. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  7. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  8. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  9. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  10. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  11. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  12. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  13. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  14. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  15. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  16. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  17. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  18. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  19. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  20. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  21. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  22. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  23. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  24. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  25. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  26. 2016Modeling Relationships in Referential Expressions with Compositional Modular Networkscited by 106
  27. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  28. 2019Zero-Shot Grounding of Objects from Natural Language Queriescited by 79
  29. 2020Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehensioncited by 50
  30. 2022Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensuscited by 49
  31. 2022Towards Robust Referring Image Segmentationcited by 49
  32. 2022BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-trainingcited by 33
  33. 2019A Real-time Global Inference Network for One-stage Referring Expression Comprehensioncited by 32
  34. 2021Detector-Free Weakly Supervised Grounding by Separationcited by 16
  35. 2023Advancing Referring Expression Segmentation Beyond Single Imagecited by 16
  36. 2022FindIt: Generalized Localization with Natural Language Queriescited by 13
  37. 2021Co-Grounding Networks with Semantic Attention for Referring Expression Comprehension in Videoscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 17; 2026: 11.

40 papers.

Sort:

By area:

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  2. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  3. 2024An Open and Comprehensive Pipeline for Unified Object Grounding and Detectioncited by 64
  4. 2023Aligning and Prompting Everything All at Once for Universal Visual Perceptioncited by 59
  5. 2024Toward Visual Grounding: A Surveycited by 52
  6. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  7. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  8. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learningcited by 29
  9. 20243D-GRES: Generalized 3D Referring Expression Segmentationcited by 18
  10. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 17
  11. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  12. 2023Generating Enhanced Negatives for Training Language-Based Object Detectorscited by 11
28 more
  1. 2025WeDetect: Fast Open-Vocabulary Object Detection as Retrievalcited by 11
  2. 2025LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generationcited by 8
  3. 2024LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Groundingcited by 7
  4. 2025Acknowledging Focus Ambiguity in Visual Questionscited by 7
  5. 2025PostAlign: Multimodal Grounding as a Corrective Lens for MLLMscited by 5
  6. 2025Re-Aligning Language to Visual Objects with an Agentic Workflowcited by 4
  7. 2025Ground-V: Teaching VLMs to Ground Complex Instructions in Pixelscited by 4
  8. 2025ReMeREC: Relation-aware and Multi-entity Referring Expression Comprehensioncited by 4
  9. 2024Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detectioncited by 3
  10. 2024LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehensioncited by 3
  11. 2025FiGO: Fine-Grained Object Counting without Annotationscited by 2
  12. 2025SORCE: Small Object Retrieval in Complex Environmentscited by 1
  13. 2025Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Groundingcited by 1
  14. 2025What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Mergingcited by 1
  15. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Modelscited by 1
  16. 2026Multi-label Instance-level Generalised Visual Grounding in Agriculturecited by 1
  17. 2026EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Modelscited by 1
  18. 2026Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perceptioncited by 1
  19. 2024Conformal-in-the-Loop for Learning with Imbalanced Noisy Datacited by 0
  20. 2025Mask-aware Text-to-Image Retrieval: Referring Expression Segmentation Meets Cross-modal Retrievalcited by 0
  21. 2025Talk in Pieces, See in Whole: Disentangled and Hierarchical Representation Learning in Language-based Object Detectioncited by 0
  22. 2026ObjEmbed: Towards Universal Multimodal Object Embeddingscited by 0
  23. 2026Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learningcited by 0
  24. 2026Consistency Beyond Contrast: Enhancing Open-Vocabulary Object Detection Robustness via Contextual Consistency Learningcited by 0
  25. 2026VISTA: Video Interaction Spatio-Temporal Analysis Benchmarkcited by 0
  26. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMscited by 0
  27. 2026UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagerycited by 0
  28. 2026Pro-Bench: Prompt-Robust Open-Vocabulary Visual Grounding Across Real-World Heterogeneous Environmentscited by 0