# Convergence and scaling of Boolean-weight optimization for hardware reservoirs

arXiv: https://arxiv.org/abs/2305.07908 · HTML: https://mlsanity.com/paper/2305.07908/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 13 May 2023.

## Builds on (most cited first, 11 of 11)

- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Reinforcement Learning in a large scale photonic Recurrent Neural Network](https://mlsanity.com/paper/1711.05133/) (2017)
- [Survey of Machine Learning Accelerators](https://mlsanity.com/paper/2009.00993/) (2020)
- [Synaptic metaplasticity in binarized neural networks](https://mlsanity.com/paper/2101.07592/) (2021)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [Probabilistic Tools for the Analysis of Randomized Optimization Heuristics](https://mlsanity.com/paper/1801.06733/) (2018)
- [Improved Iteration Complexity Bounds of Cyclic Block Coordinate Descent for Convex Problems](https://mlsanity.com/paper/1512.04680/) (2015)
- [Three dimensional waveguide-interconnects for scalable integration of photonic neural networks](https://mlsanity.com/paper/1912.08203/) (2019)
- [A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization](https://mlsanity.com/paper/1706.06493/) (2017)
- [Boolean learning under noise-perturbations in hardware neural networks](https://mlsanity.com/paper/2003.12319/) (2020)

## Abstract (arXiv)

Hardware implementation of neural network are an essential step to implement next generation efficient and powerful artificial intelligence solutions. Besides the realization of a parallel, efficient and scalable hardware architecture, the optimization of the system's extremely large parameter space with sampling-efficient approaches is essential. Here, we analytically derive the scaling laws for highly efficient Coordinate Descent applied to optimizing the readout layer of a random recurrently connection neural network, a reservoir. We demonstrate that the convergence is exponential and scales linear with the network's number of neurons. Our results perfectly reproduce the convergence and scaling of a large-scale photonic reservoir implemented in a proof-of-concept experiment. Our work therefore provides a solid foundation for such optimization in hardware networks, and identifies future directions that are promising for optimizing convergence speed during learning leveraging measures of a neural network's amplitude statistics and the weight update rule.