# A Neuromorphic Approach to Obstacle Avoidance in Robot Manipulation

arXiv: https://arxiv.org/abs/2404.05858 · HTML: https://mlsanity.com/paper/2404.05858/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 104 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 30 of 30)

- [High Speed and High Dynamic Range Video with an Event Camera](https://mlsanity.com/paper/1906.07165/) (2019)
- [Going Deeper With Directly-Trained Larger Spiking Neural Networks](https://mlsanity.com/paper/2011.05280/) (2020)
- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detection](https://mlsanity.com/paper/1903.06530/) (2019)
- [Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures](https://mlsanity.com/paper/1903.06379/) (2019)
- [Event-based Stereo Visual Odometry](https://mlsanity.com/paper/2007.15548/) (2020)
- [Fast and energy-efficient neuromorphic deep learning with first-spike times](https://mlsanity.com/paper/1912.11443/) (2019)
- [EVDodgeNet: Deep Dynamic Obstacle Dodging with Event Cameras](https://mlsanity.com/paper/1906.02919/) (2019)
- [BindsNET: A machine learning-oriented spiking neural networks library in Python](https://mlsanity.com/paper/1806.01423/) (2018)
- [Event-driven Vision and Control for UAVs on a Neuromorphic Chip](https://mlsanity.com/paper/2108.03694/) (2021)
- [Event-Driven Visual-Tactile Sensing and Learning for Robots](https://mlsanity.com/paper/2009.07083/) (2020)
- [An Introduction to Probabilistic Spiking Neural Networks: Probabilistic Models, Learning Rules, and Applications](https://mlsanity.com/paper/1910.01059/) (2019)
- [Large-Scale Neuromorphic Spiking Array Processors: A quest to mimic the brain](https://mlsanity.com/paper/1805.08932/) (2018)
- [Low-Power Neuromorphic Hardware for Signal Processing Applications](https://mlsanity.com/paper/1901.03690/) (2019)
- [Spiking Neural Networks Hardware Implementations and Challenges: a Survey](https://mlsanity.com/paper/2005.01467/) (2020)
- [Demonstrating Advantages of Neuromorphic Computation: A Pilot Study](https://mlsanity.com/paper/1811.03618/) (2018)
- [Autonomous Quadrotor Flight despite Rotor Failure with Onboard Vision Sensors: Frames vs. Events](https://mlsanity.com/paper/2102.13406/) (2021)
- [Event-based Gesture Recognition with Dynamic Background Suppression using Smartphone Computational Capabilities](https://mlsanity.com/paper/1811.07802/) (2018)
- [Small Obstacle Avoidance Based on RGB-D Semantic Segmentation](https://mlsanity.com/paper/1908.11675/) (2019)
- [Neuromorphic control for optic-flow-based landings of MAVs using the Loihi processor](https://mlsanity.com/paper/2011.00534/) (2020)
- [RGB-D-E: Event Camera Calibration for Fast 6-DOF Object Tracking](https://mlsanity.com/paper/2006.05011/) (2020)
- [Powerline Tracking with Event Cameras](https://mlsanity.com/paper/2108.00515/) (2021)
- [Introducing "Neuromorphic Computing and Engineering"](https://mlsanity.com/paper/2106.01329/) (2021)
- [Exploration of Reinforcement Learning for Event Camera using Car-like Robots](https://mlsanity.com/paper/2004.00801/) (2020)
- [Contact Anticipation for Physical Human-Robot Interaction with Robotic Manipulators using Onboard Proximity Sensors](https://mlsanity.com/paper/2110.11516/) (2021)
- [Reinforcement Learning with Low-Complexity Liquid State Machines](https://mlsanity.com/paper/1906.01695/) (2019)
- [Night vision obstacle detection and avoidance based on Bio-Inspired Vision Sensors](https://mlsanity.com/paper/2010.15509/) (2020)
- [Differential Mapping Spiking Neural Network for Sensor-Based Robot Control](https://mlsanity.com/paper/2005.10017/) (2020)
- [A Spike Learning System for Event-driven Object Recognition](https://mlsanity.com/paper/2101.08850/) (2021)
- [On-line collision avoidance for collaborative robot manipulators by adjusting off-line generated paths: An industrial use case](https://mlsanity.com/paper/1909.05159/) (2019)

## Abstract (arXiv)

Neuromorphic computing mimics computational principles of the brain in textitsilico and motivates research into event-based vision and spiking neural networks (SNNs). Event cameras (ECs) exclusively capture local intensity changes and offer superior power consumption, response latencies, and dynamic ranges. SNNs replicate biological neuronal dynamics and have demonstrated potential as alternatives to conventional artificial neural networks (ANNs), such as in reducing energy expenditure and inference time in visual classification. Nevertheless, these novel paradigms remain scarcely explored outside the domain of aerial robots. To investigate the utility of brain-inspired sensing and data processing, we developed a neuromorphic approach to obstacle avoidance on a camera-equipped manipulator. Our approach adapts high-level trajectory plans with reactive maneuvers by processing emulated event data in a convolutional SNN, decoding neural activations into avoidance motions, and adjusting plans using a dynamic motion primitive. We conducted experiments with a Kinova Gen3 arm performing simple reaching tasks that involve obstacles in sets of distinct task scenarios and in comparison to a non-adaptive baseline. Our neuromorphic approach facilitated reliable avoidance of imminent collisions in simulated and real-world experiments, where the baseline consistently failed. Trajectory adaptations had low impacts on safety and predictability criteria. Among the notable SNN properties were the correlation of computations with the magnitude of perceived motions and a robustness to different event emulation methods. Tests with a DAVIS346 EC showed similar performance, validating our experimental event emulation. Our results motivate incorporating SNN learning, utilizing neuromorphic processors, and further exploring the potential of neuromorphic methods.