# Rethinking Co-design of Neural Architectures and Hardware Accelerators

arXiv: https://arxiv.org/abs/2102.08619 · HTML: https://mlsanity.com/paper/2102.08619/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 30 of 32)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications](https://mlsanity.com/paper/1804.03230/) (2018)
- [ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation](https://mlsanity.com/paper/1812.08934/) (2018)
- [Can weight sharing outperform random architecture search? An investigation with TuNAS](https://mlsanity.com/paper/2008.06120/) (2020)
- [MobileDets: Searching for Object Detection Architectures for Mobile Accelerators](https://mlsanity.com/paper/2004.14525/) (2020)
- [Hardware/Software Co-Exploration of Neural Architectures](https://mlsanity.com/paper/1907.04650/) (2019)
- [Automatically designing CNN architectures using genetic algorithm for image classification](https://mlsanity.com/paper/1808.03818/) (2018)
- [Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks](https://mlsanity.com/paper/2002.04116/) (2020)
- [Practical Design Space Exploration](https://mlsanity.com/paper/1810.05236/) (2018)
- [Accelerator-aware Neural Network Design using AutoML](https://mlsanity.com/paper/2003.02838/) (2020)
- [Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start](https://mlsanity.com/paper/2007.09087/) (2020)
- [PyGlove: Symbolic Programming for Automated Machine Learning](https://mlsanity.com/paper/2101.08809/) (2021)
- [Synetgy: Algorithm-hardware Co-design for ConvNet Accelerators on Embedded FPGAs](https://mlsanity.com/paper/1811.08634/) (2018)
- [DANCE: Differentiable Accelerator/Network Co-Exploration](https://mlsanity.com/paper/2009.06237/) (2020)
- [Resource-Efficient Neural Architect](https://mlsanity.com/paper/1806.07912/) (2018)

## Built on by (most cited first, 12 of 12)

- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators](https://mlsanity.com/paper/2105.12842/) (2021)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Fast offset corrected in-memory training](https://mlsanity.com/paper/2303.04721/) (2023)
- [Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems](https://mlsanity.com/paper/2204.09514/) (2022)
- [Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural Architecture Search](https://mlsanity.com/paper/2210.01384/) (2022)
- [NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems](https://mlsanity.com/paper/2106.12125/) (2021)
- [QADAM: Quantization-Aware DNN Accelerator Modeling for Pareto-Optimality](https://mlsanity.com/paper/2205.13045/) (2022)
- [HALF: Holistic Auto Machine Learning for FPGAs](https://mlsanity.com/paper/2106.14771/) (2021)
- [QUIDAM: A Framework for Quantization-Aware DNN Accelerator and Model Co-Exploration](https://mlsanity.com/paper/2206.15463/) (2022)
- [Using the Abstract Computer Architecture Description Language to Model AI Hardware Accelerators](https://mlsanity.com/paper/2402.00069/) (2024)
- [HERCULES: Hardware-Efficient, Robust, Continual Learning Neural Architecture Search](https://mlsanity.com/paper/2605.04103/) (2026)

## Abstract (arXiv)

Neural architectures and hardware accelerators have been two driving forces for the progress in deep learning. Previous works typically attempt to optimize hardware given a fixed model architecture or model architecture given fixed hardware. And the dominant hardware architecture explored in this prior work is FPGAs. In our work, we target the optimization of hardware and software configurations on an industry-standard edge accelerator. We systematically study the importance and strategies of co-designing neural architectures and hardware accelerators. We make three observations: 1) the software search space has to be customized to fully leverage the targeted hardware architecture, 2) the search for the model architecture and hardware architecture should be done jointly to achieve the best of both worlds, and 3) different use cases lead to very different search outcomes. Our experiments show that the joint search method consistently outperforms previous platform-aware neural architecture search, manually crafted models, and the state-of-the-art EfficientNet on all latency targets by around 1% on ImageNet top-1 accuracy. Our method can reduce energy consumption of an edge accelerator by up to 2x under the same accuracy constraint, when co-adapting the model architecture and hardware accelerator configurations.