# Revisiting DETR Pre-training for Object Detection

arXiv: https://arxiv.org/abs/2308.01300 · HTML: https://mlsanity.com/paper/2308.01300/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2106.05095/) (2021)
- [DETRs with Hybrid Matching](https://mlsanity.com/paper/2207.13080/) (2022)
- [Aligning Pretraining for Detection via Object-Level Contrastive Learning](https://mlsanity.com/paper/2106.02637/) (2021)
- [Instance Localization for Self-supervised Detection Pretraining](https://mlsanity.com/paper/2102.08318/) (2021)
- [DETReg: Unsupervised Pretraining with Region Priors for Object Detection](https://mlsanity.com/paper/2106.04550/) (2021)
- [UniVIP: A Unified Framework for Self-Supervised Visual Pre-training](https://mlsanity.com/paper/2203.06965/) (2022)
- [CP2: Copy-Paste Contrastive Pretraining for Semantic Segmentation](https://mlsanity.com/paper/2203.11709/) (2022)
- [Group DETR v2: Strong Object Detector with Encoder-Decoder Pretraining](https://mlsanity.com/paper/2211.03594/) (2022)
- [Better Self-training for Image Classification through Self-supervision](https://mlsanity.com/paper/2109.00778/) (2021)
- [Siamese DETR](https://mlsanity.com/paper/2303.18144/) (2023)
- [Adaptive Self-Training for Object Detection](https://mlsanity.com/paper/2212.05911/) (2022)

## Built on by (most cited first, 5 of 5)

- [Rank-DETR for High Quality Object Detection](https://mlsanity.com/paper/2310.08854/) (2023)
- [Bootstrapping Autonomous Driving Radars with Self-Supervised Learning](https://mlsanity.com/paper/2312.04519/) (2023)
- [Mask Frozen-DETR: High Quality Instance Segmentation with One GPU](https://mlsanity.com/paper/2308.03747/) (2023)
- [Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers](https://mlsanity.com/paper/2403.06601/) (2024)
- [Benchmarking Deep Learning Convolutions on Energy-constrained CPUs](https://mlsanity.com/paper/2509.26217/) (2025)

## Abstract (arXiv)

Motivated by the remarkable achievements of DETR-based approaches on COCO object detection and segmentation benchmarks, recent endeavors have been directed towards elevating their performance through self-supervised pre-training of Transformers while preserving a frozen backbone. Noteworthy advancements in accuracy have been documented in certain studies. Our investigation delved deeply into a representative approach, DETReg, and its performance assessment in the context of emerging models like H-Deformable-DETR. Regrettably, DETReg proves inadequate in enhancing the performance of robust DETR-based models under full data conditions. To dissect the underlying causes, we conduct extensive experiments on COCO and PASCAL VOC probing elements such as the selection of pre-training datasets and strategies for pre-training target generation. By contrast, we employ an optimized approach named Simple Self-training which leads to marked enhancements through the combination of an improved box predictor and the Objects365 benchmark. The culmination of these endeavors results in a remarkable AP score of 59.3% on the COCO val set, outperforming H-Deformable-DETR + Swin-L without pre-training by 1.4%. Moreover, a series of synthetic pre-training datasets, generated by merging contemporary image-to-text(LLaVA) and text-to-image (SDXL) models, significantly amplifies object detection capabilities.