# Acceleration Method for Learning Fine-Layered Optical Neural Networks

arXiv: https://arxiv.org/abs/2109.01731 · HTML: https://mlsanity.com/paper/2109.01731/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 1 Sep 2021.

## Builds on (most cited first, 16 of 16)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://mlsanity.com/paper/1504.00941/) (2015)
- [Full-Capacity Unitary Recurrent Neural Networks](https://mlsanity.com/paper/1611.00035/) (2016)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs](https://mlsanity.com/paper/1612.05231/) (2016)
- [Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?](https://mlsanity.com/paper/1810.09102/) (2018)
- [Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections](https://mlsanity.com/paper/1612.00188/) (2016)
- [Wave Physics as an Analog Recurrent Neural Network](https://mlsanity.com/paper/1904.12831/) (2019)
- [Complex Gated Recurrent Neural Networks](https://mlsanity.com/paper/1806.08267/) (2018)
- [Complex Unitary Recurrent Neural Networks using Scaled Cayley Transform](https://mlsanity.com/paper/1811.04142/) (2018)
- [Design of optical neural networks with component imprecisions](https://mlsanity.com/paper/2001.01681/) (2019)
- [Fast Approximation of Rotations and Hessians matrices](https://mlsanity.com/paper/1404.7195/) (2014)
- [Matrix optimization on universal unitary photonic devices](https://mlsanity.com/paper/1808.00458/) (2018)

## Abstract (arXiv)

An optical neural network (ONN) is a promising system due to its high-speed and low-power operation. Its linear unit performs a multiplication of an input vector and a weight matrix in optical analog circuits. Among them, a circuit with a multiple-layered structure of programmable Mach-Zehnder interferometers (MZIs) can realize a specific class of unitary matrices with a limited number of MZIs as its weight matrix. The circuit is effective for balancing the number of programmable MZIs and ONN performance. However, it takes a lot of time to learn MZI parameters of the circuit with a conventional automatic differentiation (AD), which machine learning platforms are equipped with. To solve the time-consuming problem, we propose an acceleration method for learning MZI parameters. We create customized complex-valued derivatives for an MZI, exploiting Wirtinger derivatives and a chain rule. They are incorporated into our newly developed function module implemented in C++ to collectively calculate their values in a multi-layered structure. Our method is simple, fast, and versatile as well as compatible with the conventional AD. We demonstrate that our method works 20 times faster than the conventional AD when a pixel-by-pixel MNIST task is performed in a complex-valued recurrent neural network with an MZI-based hidden unit.