# MDTransformer: A Hardware-Software Co-Design of Mode-Division Photonic Transformer Accelerator with Inverse-Designed Coherent Crossbar

arXiv: https://arxiv.org/abs/2607.26016 · HTML: https://mlsanity.com/paper/2607.26016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [Photonics for artificial intelligence and neuromorphic computing](https://mlsanity.com/paper/2011.00111/) (2020)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design](https://mlsanity.com/paper/2210.09573/) (2022)
- [CrossLight: A Cross-Layer Optimized Silicon Photonic Neural Network Accelerator](https://mlsanity.com/paper/2102.06960/) (2021)
- [A Survey on Silicon Photonics for Deep Learning](https://mlsanity.com/paper/2101.01751/) (2021)
- [Lightening-Transformer: A Dynamically-operated Optically-interconnected Photonic Transformer Accelerator](https://mlsanity.com/paper/2305.19533/) (2023)
- [TRON: Transformer Neural Network Acceleration with Non-Coherent Silicon Photonics](https://mlsanity.com/paper/2303.12914/) (2023)
- [ROMANet: Fine-Grained Reuse-Driven Off-Chip Memory Access Management and Data Organization for Deep Neural Network Accelerators](https://mlsanity.com/paper/1902.10222/) (2019)
- [DRMap: A Generic DRAM Data Mapping Policy for Energy-Efficient Processing of Convolutional Neural Networks](https://mlsanity.com/paper/2004.10341/) (2020)
- [SimPhony: A Device-Circuit-Architecture Cross-Layer Modeling and Simulation Framework for Heterogeneous Electronic-Photonic AI System](https://mlsanity.com/paper/2411.13715/) (2024)
- [Characterizing Coherent Integrated Photonic Neural Networks under Imperfections](https://mlsanity.com/paper/2207.10835/) (2022)
- [PENDRAM: Enabling High-Performance and Energy-Efficient Processing of Deep Neural Networks through a Generalized DRAM Data Mapping Policy](https://mlsanity.com/paper/2408.02412/) (2024)
- [Temporal Analog Optical Computing using an On-Chip Fully Reconfigurable Photonic Signal Processor](https://mlsanity.com/paper/1712.06482/) (2017)
- [Hybrid Photonic-digital Accelerator for Attention Mechanism](https://mlsanity.com/paper/2501.11286/) (2025)

## Abstract (arXiv)

Recently, photonic transformer accelerators (PTAs) have successfully achieved significant speedup and energy efficiency improvements over electronic accelerators for expediting Transformer inference. However, state-of-the-art rely on expensive multi-wavelength light generation and large dot-product units due to active phase-shifter components, thus making their approach inefficient and impractical. To address this, we propose MDTransformer, a novel hardware-software co-design of PTA based on mode-division optical dataflow and operations. Specifically, MDTransformer performs complex matrix operations using spatial-mode interference, that leverages the inverse-designed multi-mode couplers, crossings, and Mach-Zehnder IQ modulators into a compact mode-division photonic tensor core (MPTC), capable of executing matrix multiplications in the optical domain. Its each guided mode (i.e., TE0-TE3) acts as an independent computational lane, enabling four-fold parallelism-per-waveguide without spectral filtering or free-spectral-range limitations. Moreover, its coherent detection and IQ modulation jointly encode amplitude and phase, realizing complex-valued arithmetic for full-range operations in transformers. MDTransformer offers analog multiplication with sub-4-bit effective precision and inter-modal crosstalk below -30 dB. Its inverse-designed approach also offers scalable and full compatibility with single-laser continuous-wave operation at 1550 nm. Experimental results show that MDTransformer achieves 40.4% area reduction, 63.6% power saving, 40.6% energy saving, and comparable latency over the state-of-the-art PTA across different workloads (i.e., DeiT-Tiny/Small/Base and BERT-Base/Large). These results show that MDTransformer offers a practical solution for high-performance and energy-efficient transformer-based systems.