# Parallel bandit architecture based on laser chaos for reinforcement learning

arXiv: https://arxiv.org/abs/2205.09543 · HTML: https://mlsanity.com/paper/2205.09543/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 37 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 10 of 10)

- [Massively Parallel Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1507.04296/) (2015)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Universal Quantum Control through Deep Reinforcement Learning](https://mlsanity.com/paper/1803.01857/) (2018)
- [Ultrafast photonic reinforcement learning based on laser chaos](https://mlsanity.com/paper/1704.04379/) (2017)
- [Scalable photonic reinforcement learning by time-division multiplexing of laser chaos](https://mlsanity.com/paper/1803.09425/) (2018)
- [Photonic architecture for reinforcement learning](https://mlsanity.com/paper/1907.07503/) (2019)
- [Controlling chaotic itinerancy in laser dynamics for reinforcement learning](https://mlsanity.com/paper/2205.05987/) (2022)
- [Conflict-free collective stochastic decision making by orbital angular momentum of photons through quantum interference](https://mlsanity.com/paper/2107.00877/) (2021)
- [Parameter estimation in softmax decision-making models with linear objective functions](https://mlsanity.com/paper/1502.04635/) (2015)
- [Theory of Acceleration of Decision Making by Correlated Time Sequences](https://mlsanity.com/paper/2203.16004/) (2022)

## Abstract (arXiv)

Accelerating artificial intelligence by photonics is an active field of study aiming to exploit the unique properties of photons. Reinforcement learning is an important branch of machine learning, and photonic decision-making principles have been demonstrated with respect to the multi-armed bandit problems. However, reinforcement learning could involve a massive number of states, unlike previously demonstrated bandit problems where the number of states is only one. Q-learning is a well-known approach in reinforcement learning that can deal with many states. The architecture of Q-learning, however, does not fit well photonic implementations due to its separation of update rule and the action selection. In this study, we organize a new architecture for multi-state reinforcement learning as a parallel array of bandit problems in order to benefit from photonic decision-makers, which we call parallel bandit architecture for reinforcement learning or PBRL in short. Taking a cart-pole balancing problem as an instance, we demonstrate that PBRL adapts to the environment in fewer time steps than Q-learning. Furthermore, PBRL yields faster adaptation when operated with a chaotic laser time series than the case with uniformly distributed pseudorandom numbers where the autocorrelation inherent in the laser chaos provides a positive effect. We also find that the variety of states that the system undergoes during the learning phase exhibits completely different properties between PBRL and Q-learning. The insights obtained through the present study are also beneficial for existing computing platforms, not just photonic realizations, in accelerating performances by the PBRL algorithms and correlated random sequences.