mlsanityData to 14 Oct 2026

Robust Iterative Value Conversion: Deep Reinforcement Learning for Neurochip-driven Edge Robots

arXiv 2408.13018 ↗First posted 23 Aug 2024cs.ROcited by 0

On the map: Deep reinforcement learning methods → Spiking and neuromorphic RL

Abstract (arXiv)

A neurochip is a device that reproduces the signal processing mechanisms of brain neurons and calculates Spiking Neural Networks (SNNs) with low power consumption and at high speed. Thus, neurochips are attracting attention from edge robot applications, which suffer from limited battery capacity. This paper aims to achieve deep reinforcement learning (DRL) that acquires SNN policies suitable for neurochip implementation. Since DRL requires a complex function approximation, we focus on conversion techniques from Floating Point NN (FPNN) because it is one of the most feasible SNN techniques. However, DRL requires conversions to SNNs for every policy update to collect the learning samples for a DRL-learning cycle, which updates the FPNN policy and collects the SNN policy samples. Accumulative conversion errors can significantly degrade the performance of the SNN policies. We propose Robust Iterative Value Conversion (RIVC) as a DRL that incorporates conversion error reduction and robustness to conversion errors. To reduce them, FPNN is optimized with the same number of quantization bits as an SNN. The FPNN output is not significantly changed by quantization. To robustify the conversion error, an FPNN policy that is applied with quantization is updated to increase the gap between the probability of selecting the optimal action and other actions. This step prevents unexpected replacements of the policy's optimal actions. We verified RIVC's effectiveness on a neurochip-driven robot. The results showed that RIVC consumed 1/15 times less power and increased the calculation speed by five times more than an edge CPU (quad-core ARM Cortex-A72). The previous framework with no countermeasures against conversion errors failed to train the policies. Videos from our experiments are available: https://youtu.be/Q5Z0-BvK1Tc.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spiking and neuromorphic RL, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 55 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

24 of 55 references.

References · 24

  1. 2019Surrogate Gradient Learning in Spiking Neural Networkscited by 626
  2. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  3. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  4. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  5. 2018Deep Learning in Spiking Neural Networkscited by 263
  6. 2018Direct Training for Spiking Neural Networks: Faster, Larger, Bettercited by 233
  7. 2019Spiking-YOLO: Spiking Neural Network for Energy-Efficient Object Detectioncited by 177
  8. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
  9. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  10. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  11. 2021A Free Lunch From ANN: Towards Efficient, Accurate Spiking Neural Networks Calibrationcited by 117
  12. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 98
12 more
  1. 2020Exploring the Connection Between Binary and Spiking Neural Networkscited by 38
  2. 2021Event-driven Vision and Control for UAVs on a Neuromorphic Chipcited by 34
  3. 2020BS4NN: Binarized Spiking Neural Networks with Temporal Coding and Learningcited by 24
  4. 2020Strategy and Benchmark for Converting Deep Q-Networks to Event-Driven Spiking Neural Networkscited by 23
  5. 2017Minimum Energy Quantized Neural Networkscited by 21
  6. 2020Neuromorphic control for optic-flow-based landings of MAVs using the Loihi processorcited by 13
  7. 2023Towards Efficient Verification of Quantized Neural Networkscited by 11
  8. 2020Evolved Neuromorphic Control for High Speed Divergence-based Landings of MAVscited by 8
  9. 2020RANC: Reconfigurable Architecture for Neuromorphic Computingcited by 6
  10. 2019NeuroPod: a real-time neuromorphic spiking CPG applied to roboticscited by 5
  11. 2016A Self-Driving Robot Using Deep Convolutional Neural Networks on Neuromorphic Hardwarecited by 3
  12. 2021Binarized P-Network: Deep Reinforcement Learning of Robot Control from Raw Images on FPGAcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enabling Efficient Processing of Spiking Neural Networks with On-Chip Learning on Commodity Neuromorphic Processors for Edge AI Systems
  2. 2022Towards Lossless ANN-SNN Conversion under Ultra-Low Latency with Dual-Phase Optimization
  3. 2025Hardware-Aware Fine-Tuning of Spiking Q-Networks on the SpiNNaker2 Neuromorphic Platform
  4. 2026Brain-inspired AI for Edge Intelligence: a systematic review
  5. 2025Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks
  6. 2025Autonomous Reinforcement Learning Robot Control with Intel's Loihi 2 Neuromorphic Hardware
  7. 2026Error Amplification Limits ANN-to-SNN Conversion in Continuous Control
  8. 2024MAGICS: Adversarial RL with Minimax Actors Guided by Implicit Critic Stackelberg for Convergent Neural Synthesis of Robot Safety