mlsanityData to 7 Oct 2026

Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators

arXiv 2106.06575 ↗First posted 11 Jun 2021cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

While maximizing deep neural networks' (DNNs') acceleration efficiency requires a joint search/design of three different yet highly coupled aspects, including the networks, bitwidths, and accelerators, the challenges associated with such a joint search have not yet been fully understood and addressed. The key challenges include (1) the dilemma of whether to explode the memory consumption due to the huge joint space or achieve sub-optimal designs, (2) the discrete nature of the accelerator design space that is coupled yet different from that of the networks and bitwidths, and (3) the chicken and egg problem associated with network-accelerator co-search, i.e., co-search requires operation-wise hardware cost, which is lacking during search as the optimal accelerator depending on the whole network is still unknown during search. To tackle these daunting challenges towards optimal and fast development of DNN accelerators, we propose a framework dubbed Auto-NBA to enable jointly searching for the Networks, Bitwidths, and Accelerators, by efficiently localizing the optimal design within the huge joint design space for each target dataset and acceleration specification. Our Auto-NBA integrates a heterogeneous sampling strategy to achieve unbiased search with constant memory consumption, and a novel joint-search pipeline equipped with a generic differentiable accelerator search engine. Extensive experiments and ablation studies validate that both Auto-NBA generated networks and accelerators consistently outperform state-of-the-art designs (including co-search/exploration techniques, hardware-aware NAS methods, and DNN accelerators), in terms of search time, task accuracy, and accelerator efficiency. Our codes are available at: https://github.com/RICE-EIC/Auto-NBA.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 40 of its 69 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

40 of 69 references.

References · 40

  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  3. 2019Searching for MobileNetV3cited by 1,984
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  6. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  7. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  8. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  10. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  11. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  12. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
28 more
  1. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  2. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  3. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  4. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
  5. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 98
  6. 2019Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hourscited by 87
  7. 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 79
  8. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  9. 2019AdaBits: Neural Network Quantization with Adaptive Bit-Widthscited by 63
  10. 2020ShiftAddNet: A Hardware-Inspired Deep Networkcited by 52
  11. 2019Hardware/Software Co-Exploration of Neural Architecturescited by 50
  12. 2020AutoGAN-Distiller: Searching to Compress Generative Adversarial Networkscited by 50
  13. 2020Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Taskscited by 43
  14. 2020EDD: Efficient Differentiable DNN Architecture and Implementation Co-search for Embedded AI Solutionscited by 35
  15. 2018Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutionscited by 33
  16. 2019Device-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Acceleratorscited by 32
  17. 2020Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Acceleratorcited by 30
  18. 2016Maximizing CNN Accelerator Efficiency Through Resource Partitioningcited by 29
  19. 2020AutoDNNchip: An Automated DNN Chip Predictor and Builder for Both FPGAs and ASICscited by 26
  20. 2019Adversarial Defense Through Network Profiling Based Path Extractioncited by 22
  21. 2020Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Startcited by 22
  22. 2020TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Searchcited by 21
  23. 2020SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computationcited by 16
  24. 2020Discretization-Aware Architecture Searchcited by 14
  25. 2018Joint Neural Architecture Search and Quantizationcited by 13
  26. 2020TIMELY: Pushing Data Movements and Interfaces in PIM Accelerators Towards Local and in Time Domaincited by 11
  27. 2016A Systematic Approach to Blocking Convolutional Neural Networkscited by 7
  28. 2018Design Flow of Accelerating Hybrid Extremely Low Bit-width Neural Network in Embedded FPGAcited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 2; 2023: 4; 2024: 3; 2025: 1.

12 papers.

  1. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 59
  2. 2021G-CoS: GNN-Accelerator Co-Search Towards Both Better Accuracy and Efficiencycited by 15
  3. 2024Differentiable Combinatorial Scheduling at Scalecited by 6
  4. 2022Physics-aware Differentiable Discrete Codesign for Diffractive Optical Neural Networkscited by 5
  5. 2022CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Frameworkcited by 5
  6. 2023Auto-CARD: Efficient and Robust Codec Avatar Driving for Real-time Mobile Telepresencecited by 5
  7. 2023FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Searchcited by 5
  8. 20212-in-1 Accelerator: Enabling Random Precision Switch for Winning Both Adversarial Robustness and Efficiencycited by 4
  9. 2024Trimming Down Large Spiking Vision Transformers via Heterogeneous Quantization Searchcited by 2
  10. 2025Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarkscited by 1
  11. 2023Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Explorationcited by 0
  12. 2024DCP: Learning Accelerator Dataflow for Neural Network via Propagationcited by 0