# What Makes for End-to-End Object Detection?

arXiv: https://arxiv.org/abs/2012.05780 · HTML: https://mlsanity.com/paper/2012.05780/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 30 of 49)

- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [DiffusionDet: Diffusion Model for Object Detection](https://mlsanity.com/paper/2211.09788/) (2022)
- [A Survey of Visual Transformers](https://mlsanity.com/paper/2111.06091/) (2021)
- [Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity](https://mlsanity.com/paper/2111.14330/) (2021)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Learning Object-Language Alignments for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2211.14843/) (2022)
- [Dense Distinct Query for End-to-End Object Detection](https://mlsanity.com/paper/2303.12776/) (2023)
- [Sparse Instance Activation for Real-Time Instance Segmentation](https://mlsanity.com/paper/2203.12827/) (2022)
- [AO2-DETR: Arbitrary-Oriented Object Detection Transformer](https://mlsanity.com/paper/2205.12785/) (2022)
- [Scenario Diffusion: Controllable Driving Scenario Generation With Diffusion](https://mlsanity.com/paper/2311.02738/) (2023)
- [Detection Transformer with Stable Matching](https://mlsanity.com/paper/2304.04742/) (2023)
- [ByteTrackV2: 2D and 3D Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2303.15334/) (2023)
- [Focal-PETR: Embracing Foreground for Efficient Multi-Camera 3D Object Detection](https://mlsanity.com/paper/2212.05505/) (2022)
- [Generating Driving Scenes with Diffusion](https://mlsanity.com/paper/2305.18452/) (2023)
- [Multi-clue Consistency Learning to Bridge Gaps Between General and Oriented Object in Semi-supervised Detection](https://mlsanity.com/paper/2407.05909/) (2024)
- [Mutual Supervision for Dense Object Detection](https://mlsanity.com/paper/2109.05986/) (2021)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [SAP-DETR: Bridging the Gap Between Salient Points and Queries-Based Transformer Detector for Fast Model Convergency](https://mlsanity.com/paper/2211.02006/) (2022)
- [PointScatter: Point Set Representation for Tubular Structure Extraction](https://mlsanity.com/paper/2209.05774/) (2022)
- [Consistency-Aware Anchor Pyramid Network for Crowd Localization](https://mlsanity.com/paper/2212.04067/) (2022)
- [One-to-Few Label Assignment for End-to-End Dense Detection](https://mlsanity.com/paper/2303.11567/) (2023)
- [StageInteractor: Query-based Object Detector with Cross-stage Interaction](https://mlsanity.com/paper/2304.04978/) (2023)
- [NVAutoNet: Fast and Accurate 360circ 3D Visual Perception For Self Driving](https://mlsanity.com/paper/2303.12976/) (2023)
- [HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks](https://mlsanity.com/paper/2308.12537/) (2023)
- [Visual Fault Detection of Multi-scale Key Components in Freight Trains](https://mlsanity.com/paper/2211.14522/) (2022)
- [DEYO: DETR with YOLO for Step-by-Step Object Detection](https://mlsanity.com/paper/2211.06588/) (2022)

## Abstract (arXiv)

Object detection has recently achieved a breakthrough for removing the last one non-differentiable component in the pipeline, Non-Maximum Suppression (NMS), and building up an end-to-end system. However, what makes for its one-to-one prediction has not been well understood. In this paper, we first point out that one-to-one positive sample assignment is the key factor, while, one-to-many assignment in previous detectors causes redundant predictions in inference. Second, we surprisingly find that even training with one-to-one assignment, previous detectors still produce redundant predictions. We identify that classification cost in matching cost is the main ingredient: (1) previous detectors only consider location cost, (2) by additionally introducing classification cost, previous detectors immediately produce one-to-one prediction during inference. We introduce the concept of score gap to explore the effect of matching cost. Classification cost enlarges the score gap by choosing positive samples as those of highest score in the training iteration and reducing noisy positive samples brought by only location cost. Finally, we demonstrate the advantages of end-to-end object detection on crowded scenes. The code is available at: urlhttps://github.com/PeizeSun/OneNet.