mlsanityData to 14 Oct 2026

Fast Convergence of DETR with Spatially Modulated Co-Attention

arXiv 2101.07448 ↗First posted 19 Jan 2021cs.CVcited by 176

Not on the map. Why?

Abstract (arXiv)

The recently proposed Detection Transformer (DETR) model successfully applies Transformer to objects detection and achieves comparable performance with two-stage object detection frameworks, such as Faster-RCNN. However, DETR suffers from its slow convergence. Training DETR citecarion2020end from scratch needs 500 epochs to achieve a high accuracy. To accelerate its convergence, we propose a simple yet effective scheme for improving the DETR framework, namely Spatially Modulated Co-Attention (SMCA) mechanism. The core idea of SMCA is to conduct regression-aware co-attention in DETR by constraining co-attention responses to be high near initially estimated bounding box locations. Our proposed SMCA increases DETR's convergence speed by replacing the original co-attention mechanism in the decoder while keeping other operations in DETR unchanged. Furthermore, by integrating multi-head and scale-selection attention designs into SMCA, our fully-fledged SMCA can achieve better performance compared to DETR with a dilated convolution-based backbone (45.6 mAP at 108 epochs vs. 43.3 mAP at 500 epochs). We perform extensive ablation studies on COCO dataset to validate the effectiveness of the proposed SMCA.

Key facts. Cited by 176 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 19 Jan 2021.

Builds on

36 of 50 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2020End-to-End Object Detection with Transformerscited by 7,414
  9. 2017Mask R-CNNcited by 7,277
  10. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  11. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  12. 2017Squeeze-and-Excitation Networkscited by 4,789
24 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  2. 2020Longformer: The Long-Document Transformercited by 3,413
  3. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  4. 2015Fast R-CNNcited by 2,933
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2020Reformer: The Efficient Transformercited by 1,668
  9. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  12. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  13. 2017Soft-NMS -- Improving Object Detection With One Line of Codecited by 382
  14. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 339
  15. 2016Dynamic Filter Networkscited by 319
  16. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  17. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 191
  18. 2020Rethinking Transformer-based Set Prediction for Object Detectioncited by 156
  19. 2015End-to-end people detection in crowded scenescited by 140
  20. 2018Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answeringcited by 114
  21. 2020End-to-End Object Detection with Adaptive Clustering Transformercited by 109
  22. 2019Multi-modality Latent Interaction Network for Visual Question Answeringcited by 23
  23. 2018Question-Guided Hybrid Convolution for Visual Question Answeringcited by 12
  24. 2017Recurrent Neural Networks for Semantic Instance Segmentationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 23; 2022: 65; 2023: 56; 2024: 21; 2025: 8; 2026: 3.

176 papers.

Sort:

By area:

  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  3. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  4. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  5. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 573
  6. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  7. 2023DETRs Beat YOLOs on Real-time Object Detectioncited by 533
  8. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 488
  9. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 466
  10. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  11. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  12. 2021Conditional DETR for Fast Training Convergencecited by 424
38 more
  1. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  2. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 264
  3. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  4. 2021Instances as Queriescited by 158
  5. 2022DETRs with Hybrid Matchingcited by 150
  6. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  7. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  8. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  9. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  10. 2021A Survey of Visual Transformerscited by 117
  11. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  12. 2022Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignmentcited by 98
  13. 2021FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpaintingcited by 95
  14. 2022VITA: Video Instance Segmentation via Object Token Associationcited by 94
  15. 2022MonoDETR: Depth-guided Transformer for Monocular 3D Object Detectioncited by 86
  16. 2022AdaMixer: A Fast-Converging Query-Based Object Detectorcited by 79
  17. 2023Knowing Where to Focus: Event-aware Transformer for Video Groundingcited by 79
  18. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  19. 2023iDisc: Internal Discretization for Monocular Depth Estimationcited by 73
  20. 2024DEIM: DETR with Improved Matching for Fast Convergencecited by 68
  21. 2022Accelerating DETR Convergence via Semantic-Aligned Matchingcited by 58
  22. 2023Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentationcited by 54
  23. 2022Polar Parametrization for Vision-based Surround-View 3D Detectioncited by 53
  24. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  25. 2023Rank-DETR for High Quality Object Detectioncited by 50
  26. 2022Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitationcited by 48
  27. 2023Annotation-free Audio-Visual Segmentationcited by 46
  28. 2023Detection Transformer with Stable Matchingcited by 44
  29. 2021DPT: Deformable Patch-based Transformer for Visual Recognitioncited by 43
  30. 2021VT-CLIP: Enhancing Vision-Language Models with Visual-guided Textscited by 42
  31. 2023Less is More: Focus Attention for Efficient DETRcited by 38
  32. 2021Container: Context Aggregation Networkcited by 37
  33. 2022DETRDistill: A Universal Knowledge Distillation Framework for DETR-familiescited by 36
  34. 2023V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detectioncited by 35
  35. 2024MS-DETR: Efficient DETR Training with Mixed Supervisioncited by 35
  36. 2022Towards Data-Efficient Detection Transformerscited by 34
  37. 2022Detecting and Recovering Sequential DeepFake Manipulationcited by 34
  38. 2023DilateFormer: Multi-Scale Dilated Transformer for Visual Recognitioncited by 34

50 most cited of 176 All 176 (JSON)