# Application of Transformers for Nonlinear Channel Compensation in Optical Systems

arXiv: https://arxiv.org/abs/2304.13119 · HTML: https://mlsanity.com/paper/2304.13119/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations](https://mlsanity.com/paper/1609.07061/) (2016)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [An Attention Free Transformer](https://mlsanity.com/paper/2105.14103/) (2021)
- [Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems](https://mlsanity.com/paper/2103.08212/) (2021)
- [Nonlinear Interference Mitigation via Deep Neural Networks](https://mlsanity.com/paper/1710.06234/) (2017)
- [Compensation of Fiber Nonlinearities in Digital Coherent Systems Leveraging Long Short-Term Memory Neural Networks](https://mlsanity.com/paper/2001.11802/) (2020)
- [Transfer Learning for Neural Networks-based Equalizers in Coherent Optical Systems](https://mlsanity.com/paper/2104.05081/) (2021)
- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [Ultralow complexity long short-term memory network for fiber nonlinearity mitigation in coherent optical communication systems](https://mlsanity.com/paper/2108.10212/) (2021)
- [On the Impact of Fixed Point Hardware for Optical Fiber Nonlinearity Compensation Algorithms](https://mlsanity.com/paper/1804.08545/) (2018)
- [Neural Network Architectures for Optical Channel Nonlinear Compensation in Digital Subcarrier Multiplexing Systems](https://mlsanity.com/paper/2304.06836/) (2023)

## Built on by (most cited first, 1 of 1)

- [Artificial Neural Networks for Photonic Applications: From Algorithms to Implementation](https://mlsanity.com/paper/2408.02685/) (2024)

## Abstract (arXiv)

In this paper, we introduce a new nonlinear optical channel equalizer based on Transformers. By leveraging parallel computation and attending directly to the memory across a sequence of symbols, we show that Transformers can be used effectively for nonlinear compensation (NLC) in coherent long-haul transmission systems. For this application, we present an implementation of the encoder part of the Transformer and analyze its performance over a wide range of different hyper-parameters. It is shown that by proper embeddings and processing blocks of symbols at each iteration and also carefully selecting subsets of the encoder's output to be processed together, an efficient nonlinear equalization can be achieved for different complexity constraints. To reduce the computational complexity of the attention mechanism, we further propose the use of a physic-informed mask inspired by nonlinear perturbation theory. We also compare the Transformer-NLC with digital back-propagation (DBP) under different transmission scenarios in order to demonstrate the flexibility and generalizability of the proposed data-driven solution.