# CP-DETR: Concept Prompt Guide DETR Toward Stronger Universal Object Detection

arXiv: https://arxiv.org/abs/2412.09799 · HTML: https://mlsanity.com/paper/2412.09799/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 49 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 6 of 6)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy](https://mlsanity.com/paper/2403.14610/) (2024)
- [Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Head](https://mlsanity.com/paper/2403.06892/) (2024)
- [OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network](https://mlsanity.com/paper/2209.05946/) (2022)

## Built on by (most cited first, 3 of 3)

- [DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models](https://mlsanity.com/paper/2505.24025/) (2025)
- [PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training](https://mlsanity.com/paper/2604.00503/) (2026)
- [Learning Where to Focus: Density-Driven Guidance for Detecting Dense Tiny Objects](https://mlsanity.com/paper/2512.22949/) (2025)

## Abstract (arXiv)

Recent research on universal object detection aims to introduce language in a SoTA closed-set detector and then generalize the open-set concepts by constructing large-scale (text-region) datasets for training. However, these methods face two main challenges: (i) how to efficiently use the prior information in the prompts to genericise objects and (ii) how to reduce alignment bias in the downstream tasks, both leading to sub-optimal performance in some scenarios beyond pre-training. To address these challenges, we propose a strong universal detection foundation model called CP-DETR, which is competitive in almost all scenarios, with only one pre-training weight. Specifically, we design an efficient prompt visual hybrid encoder that enhances the information interaction between prompt and visual through scale-by-scale and multi-scale fusion modules. Then, the hybrid encoder is facilitated to fully utilize the prompted information by prompt multi-label loss and auxiliary detection head. In addition to text prompts, we have designed two practical concept prompt generation methods, visual prompt and optimized prompt, to extract abstract concepts through concrete visual examples and stably reduce alignment bias in downstream tasks. With these effective designs, CP-DETR demonstrates superior universal detection performance in a broad spectrum of scenarios. For example, our Swin-T backbone model achieves 47.6 zero-shot AP on LVIS, and the Swin-L backbone model achieves 32.2 zero-shot AP on ODinW35. Furthermore, our visual prompt generation method achieves 68.4 AP on COCO val by interactive detection, and the optimized prompt achieves 73.1 fully-shot AP on ODinW13.