# Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Accelerator

arXiv: https://arxiv.org/abs/2002.05022 · HTML: https://mlsanity.com/paper/2002.05022/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2020.

## Built on by (most cited first, 30 of 30)

- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [MELTing point: Mobile Evaluation of Language Transformers](https://mlsanity.com/paper/2403.12844/) (2024)
- [A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators](https://mlsanity.com/paper/2105.12842/) (2021)
- [HAO: Hardware-aware neural Architecture Optimization for Efficient Inference](https://mlsanity.com/paper/2104.12766/) (2021)
- [G-CoS: GNN-Accelerator Co-Search Towards Both Better Accuracy and Efficiency](https://mlsanity.com/paper/2109.08983/) (2021)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators](https://mlsanity.com/paper/2106.06575/) (2021)
- [The Future of Consumer Edge-AI Computing](https://mlsanity.com/paper/2210.10514/) (2022)
- [Deep Neural Network-based Enhancement for Image and Video Streaming Systems: A Survey and Future Directions](https://mlsanity.com/paper/2106.03727/) (2021)
- [LENS: Layer Distribution Enabled Neural Architecture Search in Edge-Cloud Hierarchies](https://mlsanity.com/paper/2107.09309/) (2021)
- [CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework](https://mlsanity.com/paper/2212.03965/) (2022)
- [unzipFPGA: Enhancing FPGA-based CNN Engines with On-the-Fly Weights Generation](https://mlsanity.com/paper/2103.05600/) (2021)
- [MetaML-Pro: Cross-Stage Design Flow Automation for Efficient Deep Learning Acceleration](https://mlsanity.com/paper/2502.05850/) (2025)
- [A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06577/) (2021)
- [NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems](https://mlsanity.com/paper/2106.12125/) (2021)
- [MetaML: Automating Customizable Cross-Stage Design-Flow for Deep Learning Acceleration](https://mlsanity.com/paper/2306.08746/) (2023)
- [RHNAS: Realizable Hardware and Neural Architecture Search](https://mlsanity.com/paper/2106.09180/) (2021)
- [How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures](https://mlsanity.com/paper/2106.15021/) (2021)
- [RT-RCG: Neural Network and Accelerator Search Towards Effective and Real-time ECG Reconstruction from Intracardiac Electrograms](https://mlsanity.com/paper/2111.02569/) (2021)
- [Algorithm and Hardware Co-design for Reconfigurable CNN Accelerator](https://mlsanity.com/paper/2111.12787/) (2021)
- [Colab NAS: Obtaining lightweight task-specific convolutional neural networks following Occam's razor](https://mlsanity.com/paper/2212.07700/) (2022)
- [Tiny Classifier Circuits: Evolving Accelerators for Tabular Data](https://mlsanity.com/paper/2303.00031/) (2023)
- [TransCODE: Co-design of Transformers and Accelerators for Efficient Training and Inference](https://mlsanity.com/paper/2303.14882/) (2023)
- [Mitigating Memory Wall Effects in CNN Engines with On-the-Fly Weights Generation](https://mlsanity.com/paper/2307.13412/) (2023)
- [Using the Abstract Computer Architecture Description Language to Model AI Hardware Accelerators](https://mlsanity.com/paper/2402.00069/) (2024)
- [SONAR: Joint Architecture and System Optimization Search](https://mlsanity.com/paper/2208.12218/) (2022)
- [Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Exploration](https://mlsanity.com/paper/2301.09312/) (2023)
- [A Survey on Multi-Objective Neural Architecture Search](https://mlsanity.com/paper/2307.09099/) (2023)
- [A Survey on Design Methodologies for Accelerating Deep Learning on Heterogeneous Architectures](https://mlsanity.com/paper/2311.17815/) (2023)
- [Automated and Holistic Co-design of Neural Networks and ASICs for Enabling In-Pixel Intelligence](https://mlsanity.com/paper/2407.14560/) (2024)

## Abstract (arXiv)

Neural architecture search (NAS) has been very successful at outperforming human-designed convolutional neural networks (CNN) in accuracy, and when hardware information is present, latency as well. However, NAS-designed CNNs typically have a complicated topology, therefore, it may be difficult to design a custom hardware (HW) accelerator for such CNNs. We automate HW-CNN codesign using NAS by including parameters from both the CNN model and the HW accelerator, and we jointly search for the best model-accelerator pair that boosts accuracy and efficiency. We call this Codesign-NAS. In this paper we focus on defining the Codesign-NAS multiobjective optimization problem, demonstrating its effectiveness, and exploring different ways of navigating the codesign search space. For CIFAR-10 image classification, we enumerate close to 4 billion model-accelerator pairs, and find the Pareto frontier within that large search space. This allows us to evaluate three different reinforcement-learning-based search strategies. Finally, compared to ResNet on its most optimal HW accelerator from within our HW design space, we improve on CIFAR-100 classification accuracy by 1.3% while simultaneously increasing performance/area by 41% in just~1000 GPU-hours of running Codesign-NAS.