# SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization

arXiv: https://arxiv.org/abs/1912.05027 · HTML: https://mlsanity.com/paper/1912.05027/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2019.

## Built on by (most cited first, 30 of 50)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Bottleneck Transformers for Visual Recognition](https://mlsanity.com/paper/2101.11605/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [RTMDet: An Empirical Study of Designing Real-Time Object Detectors](https://mlsanity.com/paper/2212.07784/) (2022)
- [Dynamic Head: Unifying Object Detection Heads with Attentions](https://mlsanity.com/paper/2106.08322/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [CBNet: A Composite Backbone Network Architecture for Object Detection](https://mlsanity.com/paper/2107.00420/) (2021)
- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling](https://mlsanity.com/paper/2104.08418/) (2021)
- [HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers](https://mlsanity.com/paper/2106.06560/) (2021)
- [BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training](https://mlsanity.com/paper/2203.13249/) (2022)
- [Simple Training Strategies and Model Scaling for Object Detection](https://mlsanity.com/paper/2107.00057/) (2021)
- [Scale-aware Automatic Augmentation for Object Detection](https://mlsanity.com/paper/2103.17220/) (2021)
- [OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection](https://mlsanity.com/paper/2103.04507/) (2021)
- [GiraffeDet: A Heavy-Neck Paradigm for Object Detection](https://mlsanity.com/paper/2202.04256/) (2022)
- [MAE-DET: Revisiting Maximum Entropy Principle in Zero-Shot NAS for Efficient Object Detection](https://mlsanity.com/paper/2111.13336/) (2021)
- [Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition](https://mlsanity.com/paper/2204.04654/) (2022)
- [FBNetV5: Neural Architecture Search for Multiple Tasks in One Run](https://mlsanity.com/paper/2111.10007/) (2021)
- [Auto-scaling Vision Transformers without Training](https://mlsanity.com/paper/2202.11921/) (2022)
- [DynamicDet: A Unified Dynamic Architecture for Object Detection](https://mlsanity.com/paper/2304.05552/) (2023)
- [CAD-Estate: Large-scale CAD Model Annotation in RGB Videos](https://mlsanity.com/paper/2306.09011/) (2023)
- [The surprising impact of mask-head architecture on novel class segmentation](https://mlsanity.com/paper/2104.00613/) (2021)
- [YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs](https://mlsanity.com/paper/2110.13713/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [T-AutoML: Automated Machine Learning for Lesion Segmentation using Transformers in 3D Medical Imaging](https://mlsanity.com/paper/2111.07535/) (2021)
- [USB: Universal-Scale Object Detection Benchmark](https://mlsanity.com/paper/2103.14027/) (2021)
- [Learning Versatile Neural Architectures by Propagating Network Codes](https://mlsanity.com/paper/2103.13253/) (2021)
- [Efficient Joint-Dimensional Search with Solution Space Regularization for Real-Time Semantic Segmentation](https://mlsanity.com/paper/2208.05271/) (2022)
- [AutoFocusFormer: Image Segmentation off the Grid](https://mlsanity.com/paper/2304.12406/) (2023)
- [Benchmarking the Robustness of Instance Segmentation Models](https://mlsanity.com/paper/2109.01123/) (2021)

## Abstract (arXiv)

Convolutional neural networks typically encode an input image into a series of intermediate features with decreasing resolutions. While this structure is suited to classification tasks, it does not perform well for tasks requiring simultaneous recognition and localization (e.g., object detection). The encoder-decoder architectures are proposed to resolve this by applying a decoder network onto a backbone model designed for classification tasks. In this paper, we argue encoder-decoder architecture is ineffective in generating strong multi-scale features because of the scale-decreased backbone. We propose SpineNet, a backbone with scale-permuted intermediate features and cross-scale connections that is learned on an object detection task by Neural Architecture Search. Using similar building blocks, SpineNet models outperform ResNet-FPN models by ~3% AP at various scales while using 10-20% fewer FLOPs. In particular, SpineNet-190 achieves 52.5% AP with a MaskR-CNN detector and achieves 52.1% AP with a RetinaNet detector on COCO for a single model without test-time augmentation, significantly outperforms prior art of detectors. SpineNet can transfer to classification tasks, achieving 5% top-1 accuracy improvement on a challenging iNaturalist fine-grained dataset. Code is at: https://github.com/tensorflow/tpu/tree/master/models/official/detection.