# Photonic Reconfigurable Accelerators for Efficient Inference of CNNs with Mixed-Sized Tensors

arXiv: https://arxiv.org/abs/2207.05278 · HTML: https://mlsanity.com/paper/2207.05278/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit](https://mlsanity.com/paper/2008.11659/) (2020)
- [Dissecting the Graphcore IPU Architecture via Microbenchmarking](https://mlsanity.com/paper/1912.03413/) (2019)
- [CrossLight: A Cross-Layer Optimized Silicon Photonic Neural Network Accelerator](https://mlsanity.com/paper/2102.06960/) (2021)
- [Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights](https://mlsanity.com/paper/2007.00864/) (2020)
- [Digital Electronics and Analog Photonics for Convolutional Neural Networks (DEAP-CNNs)](https://mlsanity.com/paper/1907.01525/) (2019)
- [Photonic tensor cores for machine learning](https://mlsanity.com/paper/2002.03780/) (2020)
- [Learning on Hardware: A Tutorial on Neural Network Accelerators and Co-Processors](https://mlsanity.com/paper/2104.09252/) (2021)
- [Silicon Photonic Microring Based Chip-Scale Accelerator for Delayed Feedback Reservoir Computing](https://mlsanity.com/paper/2101.00557/) (2021)

## Built on by (most cited first, 5 of 5)

- [SCONNA: A Stochastic Computing Based Optical Accelerator for Ultra-Fast, Energy-Efficient Inference of Integer-Quantized CNNs](https://mlsanity.com/paper/2302.07036/) (2023)
- [An Optical XNOR-Bitcount Based Accelerator for Efficient Inference of Binary Neural Networks](https://mlsanity.com/paper/2302.06405/) (2023)
- [High-Speed and Energy-Efficient Non-Binary Computing with Polymorphic Electro-Optic Circuits and Architectures](https://mlsanity.com/paper/2304.07608/) (2023)
- [A Comparative Analysis of Microrings Based Incoherent Photonic GEMM Accelerators](https://mlsanity.com/paper/2402.03149/) (2024)
- [HEANA: A Hybrid Time-Amplitude Analog Optical Accelerator with Flexible Dataflows for Energy-Efficient CNN Inference](https://mlsanity.com/paper/2402.03247/) (2024)

## Abstract (arXiv)

Photonic Microring Resonator (MRR) based hardware accelerators have been shown to provide disruptive speedup and energy-efficiency improvements for processing deep Convolutional Neural Networks (CNNs). However, previous MRR-based CNN accelerators fail to provide efficient adaptability for CNNs with mixed-sized tensors. One example of such CNNs is depthwise separable CNNs. Performing inferences of CNNs with mixed-sized tensors on such inflexible accelerators often leads to low hardware utilization, which diminishes the achievable performance and energy efficiency from the accelerators. In this paper, we present a novel way of introducing reconfigurability in the MRR-based CNN accelerators, to enable dynamic maximization of the size compatibility between the accelerator hardware components and the CNN tensors that are processed using the hardware components. We classify the state-of-the-art MRR-based CNN accelerators from prior works into two categories, based on the layout and relative placements of the utilized hardware components in the accelerators. We then use our method to introduce reconfigurability in accelerators from these two classes, to consequently improve their parallelism, the flexibility of efficiently mapping tensors of different sizes, speed, and overall energy efficiency. We evaluate our reconfigurable accelerators against three prior works for the area proportionate outlook (equal hardware area for all accelerators). Our evaluation for the inference of four modern CNNs indicates that our designed reconfigurable CNN accelerators provide improvements of up to 1.8x in Frames-Per-Second (FPS) and up to 1.5x in FPS/W, compared to an MRR-based accelerator from prior work.