# AlignDet: Aligning Pre-training and Fine-tuning in Object Detection

arXiv: https://arxiv.org/abs/2307.11077 · HTML: https://mlsanity.com/paper/2307.11077/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 77 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)

## Built on by (most cited first, 14 of 14)

- [Crowd-SAM: SAM as a Smart Annotator for Object Detection in Crowded Scenes](https://mlsanity.com/paper/2407.11464/) (2024)
- [SSLChange: A Self-supervised Change Detection Framework Based on Domain Adaptation](https://mlsanity.com/paper/2405.18224/) (2024)
- [MutDet: Mutually Optimizing Pre-training for Remote Sensing Object Detection](https://mlsanity.com/paper/2407.09920/) (2024)
- [THESAURUS: Contrastive Graph Clustering by Swapping Fused Gromov-Wasserstein Couplings](https://mlsanity.com/paper/2412.11550/) (2024)
- [Unsupervised Pre-training with Language-Vision Prompts for Low-Data Instance Segmentation](https://mlsanity.com/paper/2405.13388/) (2024)
- [Bi-Level Optimization for Self-Supervised AI-Generated Face Detection](https://mlsanity.com/paper/2507.22824/) (2025)
- [Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2406.08238/) (2024)
- [Leveraging Self-Supervised Instance Contrastive Learning for Radar Object Detection](https://mlsanity.com/paper/2402.08427/) (2024)
- [Long-Tailed Object Detection Pre-training: Dynamic Rebalancing Contrastive Learning with Dual Reconstruction](https://mlsanity.com/paper/2411.09453/) (2024)
- [MedSeqFT: Sequential Fine-tuning Foundation Models for 3D Medical Image Segmentation](https://mlsanity.com/paper/2509.06096/) (2025)
- [Aligned Unsupervised Pretraining of Object Detectors with Self-training](https://mlsanity.com/paper/2307.15697/) (2023)
- [Advancing Metallic Surface Defect Detection via Anomaly-Guided Pretraining on a Large Industrial Dataset](https://mlsanity.com/paper/2509.18919/) (2025)
- [Rethinking Transfer Learning for Industrial Inspection: DINOv3 vs. ImageNet Pretraining Across RGB and X-ray Tasks](https://mlsanity.com/paper/2605.23472/) (2026)
- [SPARC: SuperPixel-Aware Region Contrastive Learning for Self-Supervised Dense Prediction](https://mlsanity.com/paper/2609.25067/) (2026)

## Abstract (arXiv)

The paradigm of large-scale pre-training followed by downstream fine-tuning has been widely employed in various object detection algorithms. In this paper, we reveal discrepancies in data, model, and task between the pre-training and fine-tuning procedure in existing practices, which implicitly limit the detector's performance, generalization ability, and convergence speed. To this end, we propose AlignDet, a unified pre-training framework that can be adapted to various existing detectors to alleviate the discrepancies. AlignDet decouples the pre-training process into two stages, i.e., image-domain and box-domain pre-training. The image-domain pre-training optimizes the detection backbone to capture holistic visual abstraction, and box-domain pre-training learns instance-level semantics and task-aware concepts to initialize the parts out of the backbone. By incorporating the self-supervised pre-trained backbones, we can pre-train all modules for various detectors in an unsupervised paradigm. As depicted in Figure 1, extensive experiments demonstrate that AlignDet can achieve significant improvements across diverse protocols, such as detection algorithm, model backbone, data setting, and training schedule. For example, AlignDet improves FCOS by 5.3 mAP, RetinaNet by 2.1 mAP, Faster R-CNN by 3.3 mAP, and DETR by 2.3 mAP under fewer epochs.