# In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Vision

arXiv: https://arxiv.org/abs/2212.10881 · HTML: https://mlsanity.com/paper/2212.10881/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Going Deeper in Spiking Neural Networks: VGG and Residual Architectures](https://mlsanity.com/paper/1802.02627/) (2018)
- [Training Deep Spiking Neural Networks using Backpropagation](https://mlsanity.com/paper/1608.08782/) (2016)
- [Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch](https://mlsanity.com/paper/2010.01729/) (2020)
- [DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks](https://mlsanity.com/paper/2008.03658/) (2020)
- [Temporal Coding in Spiking Neural Networks with Alpha Synaptic Function: Learning with Backpropagation](https://mlsanity.com/paper/1907.13223/) (2019)
- [Conversion of Artificial Recurrent Neural Networks to Spiking Neural Networks for Low-power Neuromorphic Hardware](https://mlsanity.com/paper/1601.04187/) (2016)
- [STDP-based spiking deep convolutional neural networks for object recognition](https://mlsanity.com/paper/1611.01421/) (2016)
- [P2M: A Processing-in-Pixel-in-Memory Paradigm for Resource-Constrained TinyML Applications](https://mlsanity.com/paper/2203.04737/) (2022)
- [Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays](https://mlsanity.com/paper/2004.12525/) (2020)
- [Training Energy-Efficient Deep Spiking Neural Networks with Single-Spike Hybrid Input Encoding](https://mlsanity.com/paper/2107.12374/) (2021)
- [Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking Neural Networks?](https://mlsanity.com/paper/2112.12133/) (2021)
- [Towards Low-Latency Energy-Efficient Deep SNNs via Attention-Guided Compression](https://mlsanity.com/paper/2107.12445/) (2021)
- [P2M-DeTrack: Processing-in-Pixel-in-Memory for Energy-efficient and Real-Time Multi-Object Detection and Tracking](https://mlsanity.com/paper/2205.14285/) (2022)
- [Fundamental Limits on Energy-Delay-Accuracy of In-memory Architectures in Inference Applications](https://mlsanity.com/paper/2012.13645/) (2020)
- [Toward Efficient Hyperspectral Image Processing inside Camera Pixels](https://mlsanity.com/paper/2203.05696/) (2022)
- [Hoyer regularizer is all you need for ultra low-latency spiking neural networks](https://mlsanity.com/paper/2212.10170/) (2022)
- [Towards Energy-Efficient, Low-Latency and Accurate Spiking LSTMs](https://mlsanity.com/paper/2210.12613/) (2022)

## Built on by (most cited first, 6 of 6)

- [MaskVD: Region Masking for Efficient Video Object Detection](https://mlsanity.com/paper/2407.12067/) (2024)
- [Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology](https://mlsanity.com/paper/2312.01213/) (2023)
- [Let's Roll: Synthetic Dataset Analysis for Pedestrian Detection Across Different Shutter Types](https://mlsanity.com/paper/2309.08136/) (2023)
- [Energy-Efficient & Real-Time Computer Vision with Intelligent Skipping via Reconfigurable CMOS Image Sensors](https://mlsanity.com/paper/2409.17341/) (2024)
- [When Bio-Inspired Computing meets Deep Learning: Low-Latency, Accurate, & Energy-Efficient Spiking Neural Networks from Artificial Neural Networks](https://mlsanity.com/paper/2312.06900/) (2023)
- [ReDistill: Residual Encoded Distillation for Peak Memory Reduction of CNNs](https://mlsanity.com/paper/2406.03744/) (2024)

## Abstract (arXiv)

Due to the high activation sparsity and use of accumulates (AC) instead of expensive multiply-and-accumulates (MAC), neuromorphic spiking neural networks (SNNs) have emerged as a promising low-power alternative to traditional DNNs for several computer vision (CV) applications. However, most existing SNNs require multiple time steps for acceptable inference accuracy, hindering real-time deployment and increasing spiking activity and, consequently, energy consumption. Recent works proposed direct encoding that directly feeds the analog pixel values in the first layer of the SNN in order to significantly reduce the number of time steps. Although the overhead for the first layer MACs with direct encoding is negligible for deep SNNs and the CV processing is efficient using SNNs, the data transfer between the image sensors and the downstream processing costs significant bandwidth and may dominate the total energy. To mitigate this concern, we propose an in-sensor computing hardware-software co-design framework for SNNs targeting image recognition tasks. Our approach reduces the bandwidth between sensing and processing by 12-96x and the resulting total energy by 2.32x compared to traditional CV processing, with a 3.8% reduction in accuracy on ImageNet.