# SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection

arXiv: https://arxiv.org/abs/1911.09929 · HTML: https://mlsanity.com/paper/1911.09929/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2019.

## Built on by (most cited first, 25 of 25)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [CBNet: A Composite Backbone Network Architecture for Object Detection](https://mlsanity.com/paper/2107.00420/) (2021)
- [TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search](https://mlsanity.com/paper/2105.11871/) (2021)
- [SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving](https://mlsanity.com/paper/2106.11118/) (2021)
- [G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation](https://mlsanity.com/paper/2108.07482/) (2021)
- [NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02038/) (2021)
- [Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation](https://mlsanity.com/paper/2105.12971/) (2021)
- [OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection](https://mlsanity.com/paper/2103.04507/) (2021)
- [Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelines](https://mlsanity.com/paper/2106.15397/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [FAQ: Feature Aggregated Queries for Transformer-based Video Object Detectors](https://mlsanity.com/paper/2303.08319/) (2023)
- [USB: Universal-Scale Object Detection Benchmark](https://mlsanity.com/paper/2103.14027/) (2021)
- [EAutoDet: Efficient Architecture Search for Object Detection](https://mlsanity.com/paper/2203.10747/) (2022)
- [NASOA: Towards Faster Task-oriented Online Fine-tuning with a Zoo of Models](https://mlsanity.com/paper/2108.03434/) (2021)
- [DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture](https://mlsanity.com/paper/2109.05765/) (2021)
- [NAS-FCOS: Efficient Search for Object Detection Architectures](https://mlsanity.com/paper/2110.12423/) (2021)
- [LLM-Guided Evolution: An Autonomous Model Optimization for Object Detection](https://mlsanity.com/paper/2504.02280/) (2025)
- [Automated visual inspection of silicon detectors in CMS experiment](https://mlsanity.com/paper/2206.02572/) (2022)
- [TopoNAS: Boosting Search Efficiency of Gradient-based NAS via Topological Simplification](https://mlsanity.com/paper/2408.01311/) (2024)
- [A formal definition and meta-model for a machine theory of mind](https://mlsanity.com/paper/2606.03471/) (2026)
- [Improving Differentiable Architecture Search with a Generative Model](https://mlsanity.com/paper/2112.00171/) (2021)
- [PaRT: Parallel Learning Towards Robust and Transparent AI](https://mlsanity.com/paper/2201.09534/) (2022)
- [A Survey on Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2307.09099/) (2023)
- [Underwater object detection in sonar imagery with detection transformer and Zero-shot neural architecture search](https://mlsanity.com/paper/2505.06694/) (2025)
- [CamoNAS: Neural Architecture Search for Enhanced Camouflaged Object Detection](https://mlsanity.com/paper/2607.01870/) (2026)

## Abstract (arXiv)

The state-of-the-art object detection method is complicated with various modules such as backbone, feature fusion neck, RPN and RCNN head, where each module may have different designs and structures. How to leverage the computational cost and accuracy trade-off for the structural combination as well as the modular selection of multiple modules? Neural architecture search (NAS) has shown great potential in finding an optimal solution. Existing NAS works for object detection only focus on searching better design of a single module such as backbone or feature fusion neck, while neglecting the balance of the whole system. In this paper, we present a two-stage coarse-to-fine searching strategy named Structural-to-Modular NAS (SM-NAS) for searching a GPU-friendly design of both an efficient combination of modules and better modular-level architecture for object detection. Specifically, Structural-level searching stage first aims to find an efficient combination of different modules; Modular-level searching stage then evolves each specific module and pushes the Pareto front forward to a faster task-specific network. We consider a multi-objective search where the search space covers many popular designs of detection methods. We directly search a detection backbone without pre-trained models or any proxy task by exploring a fast training from scratch strategy. The resulting architectures dominate state-of-the-art object detection systems in both inference time and accuracy and demonstrate the effectiveness on multiple detection datasets, e.g. halving the inference time with additional 1% mAP improvement compared to FPN and reaching 46% mAP with the similar inference time of MaskRCNN.