# Efficient DETR: Improving End-to-End Object Detector with Dense Prior

arXiv: https://arxiv.org/abs/2104.01318 · HTML: https://mlsanity.com/paper/2104.01318/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 44 references that are in the map.
- First posted on arXiv 3 Apr 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [CornerNet: Detecting Objects as Paired Keypoints](https://mlsanity.com/paper/1808.01244/) (2018)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection](https://mlsanity.com/paper/1912.02424/) (2019)
- [Sparse R-CNN: End-to-End Object Detection with Learnable Proposals](https://mlsanity.com/paper/2011.12450/) (2020)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [Rethinking Transformer-based Set Prediction for Object Detection](https://mlsanity.com/paper/2011.10881/) (2020)
- [Probabilistic Anchor Assignment with IoU Prediction for Object Detection](https://mlsanity.com/paper/2007.08103/) (2020)

## Built on by (most cited first, 30 of 128)

- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection](https://mlsanity.com/paper/2203.13310/) (2022)
- [AdaMixer: A Fast-Converging Query-Based Object Detector](https://mlsanity.com/paper/2203.16507/) (2022)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [Dense Distinct Query for End-to-End Object Detection](https://mlsanity.com/paper/2303.12776/) (2023)
- [DEIM: DETR with Improved Matching for Fast Convergence](https://mlsanity.com/paper/2412.04234/) (2024)
- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [FocalFormer3D : Focusing on Hard Instance for 3D Object Detection](https://mlsanity.com/paper/2308.04556/) (2023)
- [IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection](https://mlsanity.com/paper/2403.15241/) (2024)
- [Detection Transformer with Stable Matching](https://mlsanity.com/paper/2304.04742/) (2023)
- [Audio-Visual Synchronisation in the wild](https://mlsanity.com/paper/2112.04432/) (2021)
- [Object as Query: Lifting any 2D Object Detector to 3D Detection](https://mlsanity.com/paper/2301.02364/) (2023)
- [Less is More: Focus Attention for Efficient DETR](https://mlsanity.com/paper/2307.12612/) (2023)
- [Lite DETR : An Interleaved Multi-Scale Encoder for Efficient DETR](https://mlsanity.com/paper/2303.07335/) (2023)
- [MS-DETR: Efficient DETR Training with Mixed Supervision](https://mlsanity.com/paper/2401.03989/) (2024)
- [NMS Strikes Back](https://mlsanity.com/paper/2212.06137/) (2022)

## Abstract (arXiv)

The recently proposed end-to-end transformer detectors, such as DETR and Deformable DETR, have a cascade structure of stacking 6 decoder layers to update object queries iteratively, without which their performance degrades seriously. In this paper, we investigate that the random initialization of object containers, which include object queries and reference points, is mainly responsible for the requirement of multiple iterations. Based on our findings, we propose Efficient DETR, a simple and efficient pipeline for end-to-end object detection. By taking advantage of both dense detection and sparse set detection, Efficient DETR leverages dense prior to initialize the object containers and brings the gap of the 1-decoder structure and 6-decoder structure. Experiments conducted on MS COCO show that our method, with only 3 encoder layers and 1 decoder layer, achieves competitive performance with state-of-the-art object detection methods. Efficient DETR is also robust in crowded scenes. It outperforms modern detectors on CrowdHuman dataset by a large margin.