mlsanityData to 7 Oct 2026

Rethinking Transformer-based Set Prediction for Object Detection

arXiv 2011.10881 ↗First posted 21 Nov 2020cs.CV cs.LGcited by 156

Not on the map. Why?

Abstract (arXiv)

DETR is a recently proposed Transformer-based method which views object detection as a set prediction problem and achieves state-of-the-art performance but demands extra-long training time to converge. In this paper, we investigate the causes of the optimization difficulty in the training of DETR. Our examinations reveal several factors contributing to the slow convergence of DETR, primarily the issues with the Hungarian loss and the Transformer cross-attention mechanism. To overcome these issues we propose two solutions, namely, TSP-FCOS (Transformer-based Set Prediction with FCOS) and TSP-RCNN (Transformer-based Set Prediction with RCNN). Experimental results show that the proposed methods not only converge much faster than the original DETR, but also significantly outperform DETR and other baselines in terms of detection accuracy.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 61; 2023: 34; 2024: 22; 2025: 8; 2026: 2.

156 papers.

Sort:

By area:

  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  3. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  4. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  5. 2021DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queriescited by 655
  6. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  7. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 550
  8. 2022DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingcited by 486
  9. 2021Conditional DETR for Fast Training Convergencecited by 422
  10. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  11. 2021Fast Convergence of DETR with Spatially Modulated Co-Attentioncited by 176
  12. 2021Instances as Queriescited by 157
38 more
  1. 2021All Tokens Matter: Token Labeling for Training Better Vision Transformerscited by 144
  2. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  3. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  4. 2021Efficient DETR: Improving End-to-End Object Detector with Dense Priorcited by 128
  5. 2021A Survey of Visual Transformerscited by 117
  6. 2022Global Tracking Transformerscited by 110
  7. 2022AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensingcited by 107
  8. 2022Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignmentcited by 96
  9. 2022VITA: Video Instance Segmentation via Object Token Associationcited by 94
  10. 2022MonoDETR: Depth-guided Transformer for Monocular 3D Object Detectioncited by 85
  11. 2023Knowing Where to Focus: Event-aware Transformer for Video Groundingcited by 79
  12. 2022Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identificationcited by 78
  13. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 75
  14. 2022Boosting Crowd Counting via Multifaceted Attentioncited by 73
  15. 2023iDisc: Internal Discretization for Monocular Depth Estimationcited by 71
  16. 2022Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Spacecited by 66
  17. 2022Few Could Be Better Than All: Feature Sampling and Grouping for Scene Text Detectioncited by 65
  18. 2023Continual Detection Transformer for Incremental Object Detectioncited by 59
  19. 2024OPUS: Occupancy Prediction Using a Sparse Setcited by 56
  20. 2021Vision Transformer with Progressive Samplingcited by 44
  21. 2023LATR: 3D Lane Detection from Monocular Images with Transformercited by 44
  22. 2021Visual Grounding with Transformerscited by 42
  23. 2022Towards Weakly-Supervised Text Spotting using a Multi-Task Transformercited by 41
  24. 2023Less is More: Focus Attention for Efficient DETRcited by 38
  25. 2021A Survey of Self-Supervised and Few-Shot Object Detectioncited by 35
  26. 2022D-Former: A U-shaped Dilated Transformer for 3D Medical Image Segmentationcited by 35
  27. 2024MS-DETR: Efficient DETR Training with Mixed Supervisioncited by 34
  28. 2022BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-trainingcited by 33
  29. 2022NMS Strikes Backcited by 33
  30. 2022Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectivescited by 32
  31. 2023SG-Former: Self-guided Transformer with Evolving Token Reallocationcited by 32
  32. 2022Progressive End-to-End Object Detection in Crowded Scenescited by 31
  33. 2023PhenoBench -- A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domaincited by 31
  34. 2021End-to-End Trainable Multi-Instance Pose Estimation with Transformerscited by 30
  35. 2021TransLoc3D : Point Cloud based Large-scale Place Recognition using Adaptive Receptive Fieldscited by 29
  36. 2023BAM-DETR: Boundary-Aligned Moment Detection Transformer for Temporal Sentence Grounding in Videoscited by 29
  37. 2022Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learningcited by 27
  38. 2022TSRFormer: Table Structure Recognition with Transformerscited by 27

50 most cited of 156 All 156 (JSON)