# Conditional DETR V2: Efficient Detection Transformer with Box Queries

arXiv: https://arxiv.org/abs/2207.08914 · HTML: https://mlsanity.com/paper/2207.08914/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 47 references that are in the map.
- First posted on arXiv 18 Jul 2022.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision](https://mlsanity.com/paper/2106.01226/) (2021)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)

## Built on by (most cited first, 21 of 21)

- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [End-to-End 3D Dense Captioning with Vote2Cap-DETR](https://mlsanity.com/paper/2301.02508/) (2023)
- [Vote2Cap-DETR++: Decoupling Localization and Describing for End-to-End 3D Dense Captioning](https://mlsanity.com/paper/2309.02999/) (2023)
- [Less is More: Focus Attention for Efficient DETR](https://mlsanity.com/paper/2307.12612/) (2023)
- [LW-DETR: A Transformer Replacement to YOLO for Real-Time Detection](https://mlsanity.com/paper/2406.03459/) (2024)
- [MS-DETR: Efficient DETR Training with Mixed Supervision](https://mlsanity.com/paper/2401.03989/) (2024)
- [FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2301.04019/) (2023)
- [Uncovering and Quantifying Social Biases in Code Generation](https://mlsanity.com/paper/2305.15377/) (2023)
- [Interactive Segment Anything NeRF with Feature Imitation](https://mlsanity.com/paper/2305.16233/) (2023)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [D3ETR: Decoder Distillation for Detection Transformer](https://mlsanity.com/paper/2211.09768/) (2022)
- [Transformers in Small Object Detection: A Benchmark and Survey of State-of-the-Art](https://mlsanity.com/paper/2309.04902/) (2023)
- [StageInteractor: Query-based Object Detector with Cross-stage Interaction](https://mlsanity.com/paper/2304.04978/) (2023)
- [Uncovering and Categorizing Social Biases in Text-to-SQL](https://mlsanity.com/paper/2305.16253/) (2023)
- [Multi-Dimensional Hyena for Spatial Inductive Bias](https://mlsanity.com/paper/2309.13600/) (2023)
- [Parallel Sentence-Level Explanation Generation for Real-World Low-Resource Scenarios](https://mlsanity.com/paper/2302.10707/) (2023)
- [Real-time 3D Semantic Scene Completion Via Feature Aggregation and Conditioned Prediction](https://mlsanity.com/paper/2303.10967/) (2023)
- [Improving Long Text Understanding with Knowledge Distilled from Summarization Model](https://mlsanity.com/paper/2405.04955/) (2024)
- [Understanding differences in applying DETR to natural and medical images](https://mlsanity.com/paper/2405.17677/) (2024)
- [MSight: An Edge-Cloud Infrastructure-based Perception System for Connected Automated Vehicles](https://mlsanity.com/paper/2310.05290/) (2023)

## Abstract (arXiv)

In this paper, we are interested in Detection Transformer (DETR), an end-to-end object detection approach based on a transformer encoder-decoder architecture without hand-crafted postprocessing, such as NMS. Inspired by Conditional DETR, an improved DETR with fast training convergence, that presented box queries (originally called spatial queries) for internal decoder layers, we reformulate the object query into the format of the box query that is a composition of the embeddings of the reference point and the transformation of the box with respect to the reference point. This reformulation indicates the connection between the object query in DETR and the anchor box that is widely studied in Faster R-CNN. Furthermore, we learn the box queries from the image content, further improving the detection quality of Conditional DETR still with fast training convergence. In addition, we adopt the idea of axial self-attention to save the memory cost and accelerate the encoder. The resulting detector, called Conditional DETR V2, achieves better results than Conditional DETR, saves the memory cost and runs more efficiently. For example, for the DC5-ResNet-50 backbone, our approach achieves 44.8 AP with 16.4 FPS on the COCO val set and compared to Conditional DETR, it runs 1.6× faster, saves 74% of the overall memory cost, and improves 1.0 AP score.