# Simple Training Strategies and Model Scaling for Object Detection

arXiv: https://arxiv.org/abs/2107.00057 · HTML: https://mlsanity.com/paper/2107.00057/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 37 references that are in the map.
- First posted on arXiv 30 Jun 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detection](https://mlsanity.com/paper/1904.07392/) (2019)
- [Scaled-YOLOv4: Scaling Cross Stage Partial Network](https://mlsanity.com/paper/2011.08036/) (2020)

## Built on by (most cited first, 25 of 25)

- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [RbA: Segmenting Unknown Regions Rejected by All](https://mlsanity.com/paper/2211.14293/) (2022)
- [Towards Open-Vocabulary Video Instance Segmentation](https://mlsanity.com/paper/2304.01715/) (2023)
- [Unmasking Anomalies in Road-Scene Segmentation](https://mlsanity.com/paper/2307.13316/) (2023)
- [A General-Purpose Self-Supervised Model for Computational Pathology](https://mlsanity.com/paper/2308.15474/) (2023)
- [MP-Former: Mask-Piloted Transformer for Image Segmentation](https://mlsanity.com/paper/2303.07336/) (2023)
- [UniVS: Unified and Universal Video Segmentation with Prompts as Queries](https://mlsanity.com/paper/2402.18115/) (2024)
- [Proper Reuse of Image Classification Features Improves Object Detection](https://mlsanity.com/paper/2204.00484/) (2022)
- [Mask2Anomaly: Mask Transformer for Universal Open-set Segmentation](https://mlsanity.com/paper/2309.04573/) (2023)
- [Contextualized Spatio-Temporal Contrastive Learning with Self-Supervision](https://mlsanity.com/paper/2112.05181/) (2021)
- [Compositor: Bottom-up Clustering and Compositing for Robust Part and Object Segmentation](https://mlsanity.com/paper/2306.07404/) (2023)
- [Revisiting 3D ResNets for Video Recognition](https://mlsanity.com/paper/2109.01696/) (2021)
- [AutoFocusFormer: Image Segmentation off the Grid](https://mlsanity.com/paper/2304.12406/) (2023)
- [Learning with Free Object Segments for Long-Tailed Instance Segmentation](https://mlsanity.com/paper/2202.11124/) (2022)
- [MaskConver: Revisiting Pure Convolution Model for Panoptic Segmentation](https://mlsanity.com/paper/2312.06052/) (2023)
- [Mixtraining: A Better Trade-Off Between Compute and Performance](https://mlsanity.com/paper/2502.19513/) (2025)
- [Seg-VAR: Image Segmentation with Visual Autoregressive Modeling](https://mlsanity.com/paper/2511.12594/) (2025)
- [Training Strategies for Vision Transformers for Object Detection](https://mlsanity.com/paper/2304.02186/) (2023)
- [Optimizing Anchor-based Detectors for Autonomous Driving Scenes](https://mlsanity.com/paper/2208.06062/) (2022)
- [HAISTA-NET: Human Assisted Instance Segmentation Through Attention](https://mlsanity.com/paper/2305.03105/) (2023)
- [From Pixels to Objects: A Hierarchical Approach for Part and Object Segmentation Using Local and Global Aggregation](https://mlsanity.com/paper/2409.01353/) (2024)
- [Transformer-Based Dual-Optical Attention Fusion Crowd Head Point Counting and Localization Network](https://mlsanity.com/paper/2505.06937/) (2025)

## Abstract (arXiv)

The speed-accuracy Pareto curve of object detection systems have advanced through a combination of better model architectures, training and inference methods. In this paper, we methodically evaluate a variety of these techniques to understand where most of the improvements in modern detection systems come from. We benchmark these improvements on the vanilla ResNet-FPN backbone with RetinaNet and RCNN detectors. The vanilla detectors are improved by 7.7% in accuracy while being 30% faster in speed. We further provide simple scaling strategies to generate family of models that form two Pareto curves, named RetinaNet-RS and Cascade RCNN-RS. These simple rescaled detectors explore the speed-accuracy trade-off between the one-stage RetinaNet detectors and two-stage RCNN detectors. Our largest Cascade RCNN-RS models achieve 52.9% AP with a ResNet152-FPN backbone and 53.6% with a SpineNet143L backbone. Finally, we show the ResNet architecture, with three minor architectural changes, outperforms EfficientNet as the backbone for object detection and instance segmentation systems.