# Compensation of Fiber Nonlinearities in Digital Coherent Systems Leveraging Long Short-Term Memory Neural Networks

arXiv: https://arxiv.org/abs/2001.11802 · HTML: https://mlsanity.com/paper/2001.11802/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2020.

## Built on by (most cited first, 8 of 8)

- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [Performance and Complexity Analysis of bi-directional Recurrent Neural Network Models vs. Volterra Nonlinear Equalizers in Digital Coherent Systems](https://mlsanity.com/paper/2103.03832/) (2021)
- [Knowledge Distillation Applied to Optical Channel Equalization: Solving the Parallelization Problem of Recurrent Connection](https://mlsanity.com/paper/2212.04569/) (2022)
- [Fast and accurate waveform modeling of long-haul multi-channel optical fiber transmission using a hybrid model-data driven scheme](https://mlsanity.com/paper/2201.05502/) (2022)
- [Implementing Neural Network-Based Equalizers in a Coherent Optical Transmission System Using Field-Programmable Gate Arrays](https://mlsanity.com/paper/2212.04703/) (2022)
- [Neural Network Architectures for Optical Channel Nonlinear Compensation in Digital Subcarrier Multiplexing Systems](https://mlsanity.com/paper/2304.06836/) (2023)
- [Application of Transformers for Nonlinear Channel Compensation in Optical Systems](https://mlsanity.com/paper/2304.13119/) (2023)
- [Multi-Task Learning to Enhance Generalizability of Neural Network Equalizers in Coherent Optical Systems](https://mlsanity.com/paper/2307.05374/) (2023)

## Abstract (arXiv)

We introduce for the first time the utilization of Long short-term memory (LSTM) neural network architectures for the compensation of fiber nonlinearities in digital coherent systems. We conduct numerical simulations considering either C-band or O-band transmission systems for single channel and multi-channel 16-QAM modulation format with polarization multiplexing. A detailed analysis regarding the effect of the number of hidden units and the length of the word of symbols that trains the LSTM algorithm and corresponds to the considered channel memory is conducted in order to reveal the limits of LSTM based receiver with respect to performance and complexity. The numerical results show that LSTM Neural Networks can be very efficient as post processors of optical receivers which classify data that have undergone non-linear impairments in fiber and provide superior performance compared to digital back propagation, especially in the multi-channel transmission scenario. The complexity analysis shows that LSTM becomes more complex as the number of hidden units and the channel memory increase can be less complex than DBP in long distances (> 1000 km).