# On-Sensor Binarized Fully Convolutional Neural Network with A Pixel Processor Array

arXiv: https://arxiv.org/abs/2202.00836 · HTML: https://mlsanity.com/paper/2202.00836/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 16 of 16)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [R-FCN: Object Detection via Region-based Fully Convolutional Networks](https://mlsanity.com/paper/1605.06409/) (2016)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [TEyeD: Over 20 million real-world eye images with Pupil, Eyelid, and Iris 2D and 3D Segmentations, 2D and 3D Landmarks, 3D Eyeball, Gaze Vector, and Eye Movement Types](https://mlsanity.com/paper/2102.02115/) (2021)
- [TernausNetV2: Fully Convolutional Network for Instance Segmentation](https://mlsanity.com/paper/1806.00844/) (2018)
- [Fully Learnable Group Convolution for Acceleration of Deep Neural Networks](https://mlsanity.com/paper/1904.00346/) (2019)
- [Fully Embedding Fast Convolutional Networks on Pixel Processor Arrays](https://mlsanity.com/paper/2004.12525/) (2020)
- [A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays](https://mlsanity.com/paper/1909.05647/) (2019)
- [How Does Batch Normalization Help Binary Training?](https://mlsanity.com/paper/1909.09139/) (2019)
- [AnalogNet: Convolutional Neural Network Inference on Analog Focal Plane Sensor Processors](https://mlsanity.com/paper/2006.01765/) (2020)
- [Agile Reactive Navigation for A Non-Holonomic Mobile Robot Using A Pixel Processor Array](https://mlsanity.com/paper/2009.12796/) (2020)
- [Bringing A Robot Simulator to the SCAMP Vision System](https://mlsanity.com/paper/2105.10479/) (2021)
- [Direct Servo Control from In-Sensor CNN Inference with A Pixel Processor Array](https://mlsanity.com/paper/2106.07561/) (2021)

## Abstract (arXiv)

This work presents a method to implement fully convolutional neural networks (FCNs) on Pixel Processor Array (PPA) sensors, and demonstrates coarse segmentation and object localisation tasks. We design and train binarized FCN for both binary weights and activations using batchnorm, group convolution, and learnable threshold for binarization, producing networks small enough to be embedded on the focal plane of the PPA, with limited local memory resources, and using parallel elementary add/subtract, shifting, and bit operations only. We demonstrate the first implementation of an FCN on a PPA device, performing three convolution layers entirely in the pixel-level processors. We use this architecture to demonstrate inference generating heat maps for object segmentation and localisation at over 280 FPS using the SCAMP-5 PPA vision chip.