# Accelerating DETR Convergence via Semantic-Aligned Matching

arXiv: https://arxiv.org/abs/2203.06883 · HTML: https://mlsanity.com/paper/2203.06883/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 64 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Fully-Convolutional Siamese Networks for Object Tracking](https://mlsanity.com/paper/1606.09549/) (2016)
- [SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks](https://mlsanity.com/paper/1812.11703/) (2018)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [MOTR: End-to-End Multiple-Object Tracking with Transformer](https://mlsanity.com/paper/2105.03247/) (2021)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Libra R-CNN: Towards Balanced Learning for Object Detection](https://mlsanity.com/paper/1904.02701/) (2019)
- [Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking](https://mlsanity.com/paper/2103.11681/) (2021)
- [Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector](https://mlsanity.com/paper/1908.01998/) (2019)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)

## Built on by (most cited first, 30 of 58)

- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching](https://mlsanity.com/paper/2303.13076/) (2023)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [ConvMAE: Masked Convolution Meets Masked Autoencoders](https://mlsanity.com/paper/2205.03892/) (2022)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [Knowing Where to Focus: Event-aware Transformer for Video Grounding](https://mlsanity.com/paper/2308.06947/) (2023)
- [Cross Modal Transformer: Towards Fast and Robust 3D Object Detection](https://mlsanity.com/paper/2301.01283/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [Online Map Vectorization for Autonomous Driving: A Rasterization Perspective](https://mlsanity.com/paper/2306.10502/) (2023)
- [Rank-DETR for High Quality Object Detection](https://mlsanity.com/paper/2310.08854/) (2023)
- [Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitation](https://mlsanity.com/paper/2208.00219/) (2022)
- [Mask-Attention-Free Transformer for 3D Instance Segmentation](https://mlsanity.com/paper/2309.01692/) (2023)
- [Detection Transformer with Stable Matching](https://mlsanity.com/paper/2304.04742/) (2023)
- [Less is More: Focus Attention for Efficient DETR](https://mlsanity.com/paper/2307.12612/) (2023)
- [Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement](https://mlsanity.com/paper/2403.16131/) (2024)
- [FGAHOI: Fine-Grained Anchors for Human-Object Interaction Detection](https://mlsanity.com/paper/2301.04019/) (2023)
- [BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2312.00083/) (2023)
- [Cascade-DETR: Delving into High-Quality Universal Object Detection](https://mlsanity.com/paper/2307.11035/) (2023)
- [ChEX: Interactive Localization and Region Description in Chest X-rays](https://mlsanity.com/paper/2404.15770/) (2024)
- [Exploring Point-BEV Fusion for 3D Point Cloud Object Tracking with Transformer](https://mlsanity.com/paper/2208.05216/) (2022)
- [Enhanced Training of Query-Based Object Detection via Selective Query Recollection](https://mlsanity.com/paper/2212.07593/) (2022)
- [3DPPE: 3D Point Positional Encoding for Multi-Camera 3D Object Detection Transformers](https://mlsanity.com/paper/2211.14710/) (2022)
- [MS-DETR: Multispectral Pedestrian Detection Transformer with Loosely Coupled Fusion and Modality-Balanced Optimization](https://mlsanity.com/paper/2302.00290/) (2023)
- [Surround-View Vision-based 3D Detection for Autonomous Driving: A Survey](https://mlsanity.com/paper/2302.06650/) (2023)
- [AlignDet: Aligning Pre-training and Fine-tuning in Object Detection](https://mlsanity.com/paper/2307.11077/) (2023)
- [KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling](https://mlsanity.com/paper/2211.08071/) (2022)
- [Towards Efficient Use of Multi-Scale Features in Transformer-Based Object Detectors](https://mlsanity.com/paper/2208.11356/) (2022)
- [SAP-DETR: Bridging the Gap Between Salient Points and Queries-Based Transformer Detector for Fast Model Convergency](https://mlsanity.com/paper/2211.02006/) (2022)
- [One-to-Few Label Assignment for End-to-End Dense Detection](https://mlsanity.com/paper/2303.11567/) (2023)
- [Decoupled DETR: Spatially Disentangling Localization and Classification for Improved End-to-End Object Detection](https://mlsanity.com/paper/2310.15955/) (2023)

## Abstract (arXiv)

The recently developed DEtection TRansformer (DETR) establishes a new object detection paradigm by eliminating a series of hand-crafted components. However, DETR suffers from extremely slow convergence, which increases the training cost significantly. We observe that the slow convergence is largely attributed to the complication in matching object queries with target features in different feature embedding spaces. This paper presents SAM-DETR, a Semantic-Aligned-Matching DETR that greatly accelerates DETR's convergence without sacrificing its accuracy. SAM-DETR addresses the convergence issue from two perspectives. First, it projects object queries into the same embedding space as encoded image features, where the matching can be accomplished efficiently with aligned semantics. Second, it explicitly searches salient points with the most discriminative features for semantic-aligned matching, which further speeds up the convergence and boosts detection accuracy as well. Being like a plug and play, SAM-DETR complements existing convergence solutions well yet only introduces slight computational overhead. Extensive experiments show that the proposed SAM-DETR achieves superior convergence as well as competitive detection accuracy. The implementation codes are available at https://github.com/ZhangGongjie/SAM-DETR.