# Streamlined optical training of large-scale modern deep learning architectures with direct feedback alignment

arXiv: https://arxiv.org/abs/2409.12965 · HTML: https://mlsanity.com/paper/2409.12965/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 26 references that are in the map.
- First posted on arXiv 1 Sep 2024.

## Builds on (most cited first, 9 of 9)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Optical Transformers](https://mlsanity.com/paper/2302.10360/) (2023)
- [Photonic Differential Privacy with Direct Feedback Alignment](https://mlsanity.com/paper/2106.03645/) (2021)

## Built on by (most cited first, 3 of 3)

- [Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layers](https://mlsanity.com/paper/2509.15113/) (2025)
- [Conditioned Direct Feedback Alignment via Activity and Error Geometry](https://mlsanity.com/paper/2607.18574/) (2026)
- [What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility](https://mlsanity.com/paper/2606.21126/) (2026)

## Abstract (arXiv)

Modern deep learning relies nearly exclusively on dedicated electronic hardware accelerators. Photonic approaches, with low consumption and high operation speed, are increasingly considered for inference but, to date, remain mostly limited to relatively basic tasks. Simultaneously, the problem of training deep and complex neural networks, overwhelmingly performed through backpropagation, remains a significant limitation to the size and, consequently, the performance of current architectures and a major compute and energy bottleneck. Here, we experimentally implement a versatile and scalable training algorithm, called direct feedback alignment, on a hybrid electronic-photonic platform. An optical processing unit performs large-scale random matrix multiplications, which is the central operation of this algorithm, at speeds up to 1500 TeraOPS under 30 Watts of power. We perform optical training of modern deep learning architectures, including Transformers, with more than 1B parameters, and obtain good performances on language, vision, and diffusion-based generative tasks. We study the scaling of the training time, and demonstrate a potential advantage of our hybrid opto-electronic approach for ultra-deep and wide neural networks, thus opening a promising route to sustain the exponential growth of modern artificial intelligence beyond traditional von Neumann approaches.