# Hyperspectral In-Memory Computing with Optical Frequency Combs and Programmable Optical Memories

arXiv: https://arxiv.org/abs/2310.11014 · HTML: https://mlsanity.com/paper/2310.11014/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 77 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit](https://mlsanity.com/paper/2008.11659/) (2020)
- [Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators](https://mlsanity.com/paper/2302.08469/) (2023)
- [The physics of optical computing](https://mlsanity.com/paper/2308.00088/) (2023)
- [An optical neural network using less than 1 photon per multiplication](https://mlsanity.com/paper/2104.13467/) (2021)
- [Massively Parallel Amplitude-Only Fourier Neural Network](https://mlsanity.com/paper/2008.05853/) (2020)
- [Single-Shot Optical Neural Network](https://mlsanity.com/paper/2205.09103/) (2022)
- [Analog Iterative Machine (AIM): using light to solve quadratic optimization problems with mixed variables](https://mlsanity.com/paper/2304.12594/) (2023)
- [In-memory photonic dot-product engine with electrically programmable weight banks](https://mlsanity.com/paper/2304.14302/) (2023)

## Abstract (arXiv)

The rapid advancements in machine learning across numerous industries have amplified the demand for extensive matrix-vector multiplication operations, thereby challenging the capacities of traditional von Neumann computing architectures. To address this, researchers are currently exploring alternatives such as in-memory computing systems to develop faster and more energy-efficient hardware. In particular, there is renewed interest in computing systems based on optics, which could potentially handle matrix-vector multiplication in a more energy-efficient way. Despite promising initial results, developing a highly parallel, programmable, and scalable optical computing system capable of rivaling electronic computing hardware still remains elusive. In this context, we propose a hyperspectral in-memory computing architecture that integrates space multiplexing with frequency multiplexing of optical frequency combs and uses spatial light modulators as a programmable optical memory, thereby boosting the computational throughput and the energy efficiency. We have experimentally demonstrated multiply-accumulate operations with higher than 4-bit precision in both matrix-vector and matrix-matrix multiplications, which suggests the system's potential for a wide variety of deep learning and optimization tasks. This system exhibits extraordinary modularity, scalability, and programmability, effectively transcending the traditional limitations of optics-based computing architectures. Our approach demonstrates the potential to scale beyond peta operations per second, marking a significant step towards achieving high-throughput energy-efficient optical computing.