# Energy-Efficient Visual Search by Eye Movement and Low-Latency Spiking Neural Network

arXiv: https://arxiv.org/abs/2310.06578 · HTML: https://mlsanity.com/paper/2310.06578/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 100 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks](https://mlsanity.com/paper/1706.02609/) (2017)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks](https://mlsanity.com/paper/2007.05785/) (2020)
- [Kornia: an Open Source Differentiable Computer Vision Library for PyTorch](https://mlsanity.com/paper/1910.02190/) (2019)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks](https://mlsanity.com/paper/2303.04347/) (2023)
- [Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control](https://mlsanity.com/paper/2010.09635/) (2020)
- [Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules](https://mlsanity.com/paper/2006.16981/) (2020)
- [Saccader: Improving Accuracy of Hard Attention Models for Vision](https://mlsanity.com/paper/1908.07644/) (2019)
- [A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/2104.10719/) (2021)
- [Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networks](https://mlsanity.com/paper/2009.14456/) (2020)
- [Backpropagation with Biologically Plausible Spatio-Temporal Adjustment For Training Deep Spiking Neural Networks](https://mlsanity.com/paper/2110.08858/) (2021)
- [Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision](https://mlsanity.com/paper/2112.03423/) (2021)
- [Low-Power Neuromorphic Hardware for Signal Processing Applications](https://mlsanity.com/paper/1901.03690/) (2019)
- [Deep Reinforcement Learning with Spiking Q-learning](https://mlsanity.com/paper/2201.09754/) (2022)
- [One Timestep is All You Need: Training Spiking Neural Networks with Ultra Low Latency](https://mlsanity.com/paper/2110.05929/) (2021)
- [Object Detection Through Exploration With A Foveated Visual Field](https://mlsanity.com/paper/1408.0814/) (2014)
- [A Probabilistic Hard Attention Model For Sequentially Observed Scenes](https://mlsanity.com/paper/2111.07534/) (2021)
- [Consistency driven Sequential Transformers Attention Model for Partially Observable Scenes](https://mlsanity.com/paper/2204.00656/) (2022)
- [RetinaNet Object Detector based on Analog-to-Spiking Neural Network Conversion](https://mlsanity.com/paper/2106.05624/) (2021)
- [Learning Fixation Point Strategy for Object Detection and Classification](https://mlsanity.com/paper/1712.06897/) (2017)
- [GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction](https://mlsanity.com/paper/2210.13605/) (2022)

## Abstract (arXiv)

Human vision incorporates non-uniform resolution retina, efficient eye movement strategy, and spiking neural network (SNN) to balance the requirements in visual field size, visual resolution, energy cost, and inference latency. These properties have inspired interest in developing human-like computer vision. However, existing models haven't fully incorporated the three features of human vision, and their learned eye movement strategies haven't been compared with human's strategy, making the models' behavior difficult to interpret. Here, we carry out experiments to examine human visual search behaviors and establish the first SNN-based visual search model. The model combines an artificial retina with spiking feature extraction, memory, and saccade decision modules, and it employs population coding for fast and efficient saccade decisions. The model can learn either a human-like or a near-optimal fixation strategy, outperform humans in search speed and accuracy, and achieve high energy efficiency through short saccade decision latency and sparse activation. It also suggests that the human search strategy is suboptimal in terms of search speed. Our work connects modeling of vision in neuroscience and machine learning and sheds light on developing more energy-efficient computer vision algorithms.