# Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems

arXiv: https://arxiv.org/abs/2103.08212 · HTML: https://mlsanity.com/paper/2103.08212/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2021.

## Built on by (most cited first, 19 of 19)

- [3D Occupancy Prediction with Low-Resolution Queries via Prototype-aware View Transformation](https://mlsanity.com/paper/2503.15185/) (2025)
- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [Domain Adaptation: the Key Enabler of Neural Network Equalizers in Coherent Optical Systems](https://mlsanity.com/paper/2202.12689/) (2022)
- [Towards FPGA Implementation of Neural Network-Based Nonlinearity Mitigation Equalizers in Coherent Optical Transmission Systems](https://mlsanity.com/paper/2206.12180/) (2022)
- [Model-based Deep Learning Receiver Design for Rate-Splitting Multiple Access](https://mlsanity.com/paper/2205.00849/) (2022)
- [Spiking Neural Network Decision Feedback Equalization](https://mlsanity.com/paper/2211.04756/) (2022)
- [Knowledge Distillation Applied to Optical Channel Equalization: Solving the Parallelization Problem of Recurrent Connection](https://mlsanity.com/paper/2212.04569/) (2022)
- [Experimental Evaluation of Computational Complexity for Different Neural Network Equalizers in Optical Communications](https://mlsanity.com/paper/2109.08711/) (2021)
- [Few-bit Quantization of Neural Networks for Nonlinearity Mitigation in a Fiber Transmission Experiment](https://mlsanity.com/paper/2205.11284/) (2022)
- [Computational Complexity Evaluation of Neural Network Applications in Signal Processing](https://mlsanity.com/paper/2206.12191/) (2022)
- [Implementing Neural Network-Based Equalizers in a Coherent Optical Transmission System Using Field-Programmable Gate Arrays](https://mlsanity.com/paper/2212.04703/) (2022)
- [Neural Network Architectures for Optical Channel Nonlinear Compensation in Digital Subcarrier Multiplexing Systems](https://mlsanity.com/paper/2304.06836/) (2023)
- [Application of Transformers for Nonlinear Channel Compensation in Optical Systems](https://mlsanity.com/paper/2304.13119/) (2023)
- [Machine Learning in Short-Reach Optical Systems: A Comprehensive Survey](https://mlsanity.com/paper/2405.09557/) (2024)
- [Reservoir Computing-based Multi-Symbol Equalization for PAM 4 Short-reach Transmission](https://mlsanity.com/paper/2212.00738/) (2022)
- [Hardware Realization of Nonlinear Activation Functions for NN-based Optical Equalizers](https://mlsanity.com/paper/2305.09495/) (2023)
- [Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments](https://mlsanity.com/paper/2310.05950/) (2023)
- [Designing deep neural networks for driver intention recognition](https://mlsanity.com/paper/2402.05150/) (2024)
- [TreeNet: Layered Decision Ensembles](https://mlsanity.com/paper/2510.09654/) (2025)

## Abstract (arXiv)

We present the results of the comparative analysis of the performance versus complexity for several types of artificial neural networks (NNs) used for nonlinear channel equalization in coherent optical communication systems. The comparison has been carried out using an experimental set-up with transmission dominated by the Kerr nonlinearity and component imperfections. For the first time, we investigate the application to the channel equalization of the convolution layer (CNN) in combination with a bidirectional long short-term memory (biLSTM) layer and the design combining CNN with a multi-layer perceptron. Their performance is compared with the one delivered by the previously proposed NN equalizer models: one biLSTM layer, three-dense-layer perceptron, and the echo state network. Importantly, all architectures have been initially optimized by a Bayesian optimizer. We present the derivation of the computational complexity associated with each NN type -- in terms of real multiplications per symbol so that these results can be applied to a large number of communication systems. We demonstrated that in the specific considered experimental system the convolutional layer coupled with the biLSTM (CNN+biLSTM) provides the highest Q-factor improvement compared to the reference linear chromatic dispersion compensation (2.9 dB improvement). We examine the trade-off between the computational complexity and performance of all equalizers and demonstrate that the CNN+biLSTM is the best option when the computational complexity is not constrained, while when we restrict the complexity to lower levels, the three-layer perceptron provides the best performance. Our complexity analysis for different NNs is generic and can be applied in a wide range of physical and engineering systems.