# Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators

arXiv: https://arxiv.org/abs/2106.06575 · HTML: https://mlsanity.com/paper/2106.06575/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 69 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 30 of 40)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions](https://mlsanity.com/paper/2004.05565/) (2020)
- [Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.00090/) (2018)
- [APQ: Joint Search for Network Architecture, Pruning and Quantization Policy](https://mlsanity.com/paper/2006.08509/) (2020)
- [Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours](https://mlsanity.com/paper/1904.02877/) (2019)
- [Rethinking Differentiable Search for Mixed-Precision Neural Networks](https://mlsanity.com/paper/2004.05795/) (2020)
- [DSNAS: Direct Neural Architecture Search without Parameter Retraining](https://mlsanity.com/paper/2002.09128/) (2020)
- [AdaBits: Neural Network Quantization with Adaptive Bit-Widths](https://mlsanity.com/paper/1912.09666/) (2019)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [Hardware/Software Co-Exploration of Neural Architectures](https://mlsanity.com/paper/1907.04650/) (2019)
- [AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks](https://mlsanity.com/paper/2006.08198/) (2020)
- [Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks](https://mlsanity.com/paper/2002.04116/) (2020)
- [EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutions](https://mlsanity.com/paper/2005.02563/) (2020)
- [Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions](https://mlsanity.com/paper/1806.09228/) (2018)
- [Device-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Accelerators](https://mlsanity.com/paper/1911.00139/) (2019)
- [Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Accelerator](https://mlsanity.com/paper/2002.05022/) (2020)
- [Maximizing CNN Accelerator Efficiency Through Resource Partitioning](https://mlsanity.com/paper/1607.00064/) (2016)

## Built on by (most cited first, 12 of 12)

- [GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models](https://mlsanity.com/paper/2309.10730/) (2023)
- [G-CoS: GNN-Accelerator Co-Search Towards Both Better Accuracy and Efficiency](https://mlsanity.com/paper/2109.08983/) (2021)
- [FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search](https://mlsanity.com/paper/2308.03290/) (2023)
- [Differentiable Combinatorial Scheduling at Scale](https://mlsanity.com/paper/2406.06593/) (2024)
- [2-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiency](https://mlsanity.com/paper/2109.05223/) (2021)
- [Physics-aware Differentiable Discrete Codesign for Diffractive Optical Neural Networks](https://mlsanity.com/paper/2209.14252/) (2022)
- [CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework](https://mlsanity.com/paper/2212.03965/) (2022)
- [Auto-CARD: Efficient and Robust Codec Avatar Driving for Real-time Mobile Telepresence](https://mlsanity.com/paper/2304.11835/) (2023)
- [Trimming Down Large Spiking Vision Transformers via Heterogeneous Quantization Search](https://mlsanity.com/paper/2412.05505/) (2024)
- [Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarks](https://mlsanity.com/paper/2505.15631/) (2025)
- [Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Exploration](https://mlsanity.com/paper/2301.09312/) (2023)
- [DCP: Learning Accelerator Dataflow for Neural Network via Propagation](https://mlsanity.com/paper/2410.06553/) (2024)

## Abstract (arXiv)

While maximizing deep neural networks' (DNNs') acceleration efficiency requires a joint search/design of three different yet highly coupled aspects, including the networks, bitwidths, and accelerators, the challenges associated with such a joint search have not yet been fully understood and addressed. The key challenges include (1) the dilemma of whether to explode the memory consumption due to the huge joint space or achieve sub-optimal designs, (2) the discrete nature of the accelerator design space that is coupled yet different from that of the networks and bitwidths, and (3) the chicken and egg problem associated with network-accelerator co-search, i.e., co-search requires operation-wise hardware cost, which is lacking during search as the optimal accelerator depending on the whole network is still unknown during search. To tackle these daunting challenges towards optimal and fast development of DNN accelerators, we propose a framework dubbed Auto-NBA to enable jointly searching for the Networks, Bitwidths, and Accelerators, by efficiently localizing the optimal design within the huge joint design space for each target dataset and acceleration specification. Our Auto-NBA integrates a heterogeneous sampling strategy to achieve unbiased search with constant memory consumption, and a novel joint-search pipeline equipped with a generic differentiable accelerator search engine. Extensive experiments and ablation studies validate that both Auto-NBA generated networks and accelerators consistently outperform state-of-the-art designs (including co-search/exploration techniques, hardware-aware NAS methods, and DNN accelerators), in terms of search time, task accuracy, and accelerator efficiency. Our codes are available at: https://github.com/RICE-EIC/Auto-NBA.