# NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks

arXiv: https://arxiv.org/abs/2210.13361 · HTML: https://mlsanity.com/paper/2210.13361/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions](https://mlsanity.com/paper/2004.05565/) (2020)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions](https://mlsanity.com/paper/1711.08141/) (2017)
- [AdderNet: Do We Really Need Multiplications in Deep Learning?](https://mlsanity.com/paper/1912.13200/) (2019)
- [DeepShift: Towards Multiplication-Less Neural Networks](https://mlsanity.com/paper/1905.13298/) (2019)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [Maximizing CNN Accelerator Efficiency Through Resource Partitioning](https://mlsanity.com/paper/1607.00064/) (2016)
- [AutoDNNchip: An Automated DNN Chip Predictor and Builder for Both FPGAs and ASICs](https://mlsanity.com/paper/2001.03535/) (2020)
- [DNN-Chip Predictor: An Analytical Performance Predictor for DNN Accelerators with Various Dataflows and Hardware Architectures](https://mlsanity.com/paper/2002.11270/) (2020)
- [SmartExchange: Trading Higher-cost Memory Storage/Access for Lower-cost Computation](https://mlsanity.com/paper/2005.03403/) (2020)
- [AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence](https://mlsanity.com/paper/2101.10015/) (2021)
- [Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices](https://mlsanity.com/paper/1807.07928/) (2018)

## Built on by (most cited first, 5 of 5)

- [ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization](https://mlsanity.com/paper/2406.05981/) (2024)
- [P2-ViT: Power-of-Two Post-Training Quantization and Acceleration for Fully Quantized Vision Transformer](https://mlsanity.com/paper/2405.19915/) (2024)
- [ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation](https://mlsanity.com/paper/2407.02881/) (2024)
- [NASH: Neural Architecture and Accelerator Search for Multiplication-Reduced Hybrid Models](https://mlsanity.com/paper/2409.04829/) (2024)
- [A Benchmark on Directed Graph Representation Learning in Hardware Designs](https://mlsanity.com/paper/2410.06460/) (2024)

## Abstract (arXiv)

Multiplication is arguably the most cost-dominant operation in modern deep neural networks (DNNs), limiting their achievable efficiency and thus more extensive deployment in resource-constrained applications. To tackle this limitation, pioneering works have developed handcrafted multiplication-free DNNs, which require expert knowledge and time-consuming manual iteration, calling for fast development tools. To this end, we propose a Neural Architecture Search and Acceleration framework dubbed NASA, which enables automated multiplication-reduced DNN development and integrates a dedicated multiplication-reduced accelerator for boosting DNNs' achievable efficiency. Specifically, NASA adopts neural architecture search (NAS) spaces that augment the state-of-the-art one with hardware-inspired multiplication-free operators, such as shift and adder, armed with a novel progressive pretrain strategy (PGP) together with customized training recipes to automatically search for optimal multiplication-reduced DNNs; On top of that, NASA further develops a dedicated accelerator, which advocates a chunk-based template and auto-mapper dedicated for NASA-NAS resulting DNNs to better leverage their algorithmic properties for boosting hardware efficiency. Experimental results and ablation studies consistently validate the advantages of NASA's algorithm-hardware co-design framework in terms of achievable accuracy and efficiency tradeoffs. Codes are available at https://github.com/GATECH-EIC/NASA.