# Automated and Holistic Co-design of Neural Networks and ASICs for Enabling In-Pixel Intelligence

arXiv: https://arxiv.org/abs/2407.14560 · HTML: https://mlsanity.com/paper/2407.14560/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 42 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors](https://mlsanity.com/paper/2006.10159/) (2020)
- [Hardware/Software Co-Exploration of Neural Architectures](https://mlsanity.com/paper/1907.04650/) (2019)
- [Machine Learning in High Energy Physics Community White Paper](https://mlsanity.com/paper/1807.02876/) (2018)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference](https://mlsanity.com/paper/2102.11289/) (2021)
- [Best of Both Worlds: AutoML Codesign of a CNN and its Hardware Accelerator](https://mlsanity.com/paper/2002.05022/) (2020)
- [A reconfigurable neural network ASIC for detector front-end data compression at the HL-LHC](https://mlsanity.com/paper/2105.01683/) (2021)
- [EdgeDRNN: Recurrent Neural Network Accelerator for Edge Inference](https://mlsanity.com/paper/2012.13600/) (2020)
- [End-to-end codesign of Hessian-aware quantized neural networks for FPGAs and ASICs](https://mlsanity.com/paper/2304.06745/) (2023)

## Abstract (arXiv)

Extreme edge-AI systems, such as those in readout ASICs for radiation detection, must operate under stringent hardware constraints such as micron-level dimensions, sub-milliwatt power, and nanosecond-scale speed while providing clear accuracy advantages over traditional architectures. Finding ideal solutions means identifying optimal AI and ASIC design choices from a design space that has explosively expanded during the merger of these domains, creating non-trivial couplings which together act upon a small set of solutions as constraints tighten. It is impractical, if not impossible, to manually determine ideal choices among possibilities that easily exceed billions even in small-size problems. Existing methods to bridge this gap have leveraged theoretical understanding of hardware to f architecture search. However, the assumptions made in computing such theoretical metrics are too idealized to provide sufficient guidance during the difficult search for a practical implementation. Meanwhile, theoretical estimates for many other crucial metrics (like delay) do not even exist and are similarly variable, dependent on parameters of the process design kit (PDK). To address these challenges, we present a study that employs intelligent search using multi-objective Bayesian optimization, integrating both neural network search and ASIC synthesis in the loop. This approach provides reliable feedback on the collective impact of all cross-domain design choices. We showcase the effectiveness of our approach by finding several Pareto-optimal design choices for effective and efficient neural networks that perform real-time feature extraction from input pulses within the individual pixels of a readout ASIC.