# MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering

arXiv: https://arxiv.org/abs/2502.16486 · HTML: https://mlsanity.com/paper/2502.16486/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 33 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [YOLO-World: Real-Time Open-Vocabulary Object Detection](https://mlsanity.com/paper/2401.17270/) (2024)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [Modeling Context Between Objects for Referring Expression Understanding](https://mlsanity.com/paper/1608.00525/) (2016)
- [Exploiting Unlabeled Data with Vision and Language Models for Object Detection](https://mlsanity.com/paper/2207.08954/) (2022)
- [NExT-Chat: An LMM for Chat, Detection and Segmentation](https://mlsanity.com/paper/2311.04498/) (2023)
- [A Survey on Object Detection in Optical Remote Sensing Images](https://mlsanity.com/paper/1603.06201/) (2016)
- [Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models](https://mlsanity.com/paper/2406.16866/) (2024)
- [Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models](https://mlsanity.com/paper/2311.14552/) (2023)
- [Open Vocabulary Object Detection with Proposal Mining and Prediction Equalization](https://mlsanity.com/paper/2206.11134/) (2022)
- [TaskCLIP: Extend Large Vision-Language Model for Task Oriented Object Detection](https://mlsanity.com/paper/2403.08108/) (2024)
- [Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Head](https://mlsanity.com/paper/2403.06892/) (2024)
- [Multimodal Question Answering for Unified Information Extraction](https://mlsanity.com/paper/2310.03017/) (2023)

## Abstract (arXiv)

Open-vocabulary detection (OVD) is a challenging task to detect and classify objects from an unrestricted set of categories, including those unseen during training. Existing open-vocabulary detectors are limited by complex visual-textual misalignment and long-tailed category imbalances, leading to suboptimal performance in challenging scenarios. To address these limitations, we introduce MQADet, a universal paradigm for enhancing existing open-vocabulary detectors by leveraging the cross-modal reasoning capabilities of multimodal large language models (MLLMs). MQADet functions as a plug-and-play solution that integrates seamlessly with pre-trained object detectors without substantial additional training costs. Specifically, we design a novel three-stage Multimodal Question Answering (MQA) pipeline to guide the MLLMs to precisely localize complex textual and visual targets while effectively enhancing the focus of existing object detectors on relevant objects. To validate our approach, we present a new benchmark for evaluating our paradigm on four challenging open-vocabulary datasets, employing three state-of-the-art object detectors as baselines. Experimental results demonstrate that our proposed paradigm significantly improves the performance of existing detectors, particularly in unseen complex categories, across diverse and challenging scenarios. To facilitate future research, we will publicly release our code.