# Performance and Complexity Analysis of bi-directional Recurrent Neural Network Models vs. Volterra Nonlinear Equalizers in Digital Coherent Systems

arXiv: https://arxiv.org/abs/2103.03832 · HTML: https://mlsanity.com/paper/2103.03832/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 33 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 6 of 6)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Nonlinear Interference Mitigation via Deep Neural Networks](https://mlsanity.com/paper/1710.06234/) (2017)
- [Compensation of Fiber Nonlinearities in Digital Coherent Systems Leveraging Long Short-Term Memory Neural Networks](https://mlsanity.com/paper/2001.11802/) (2020)

## Built on by (most cited first, 6 of 6)

- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [Master memory function for delay-based reservoir computers with single-variable dynamics](https://mlsanity.com/paper/2108.12643/) (2021)
- [Knowledge Distillation Applied to Optical Channel Equalization: Solving the Parallelization Problem of Recurrent Connection](https://mlsanity.com/paper/2212.04569/) (2022)
- [Computational Complexity Evaluation of Neural Network Applications in Signal Processing](https://mlsanity.com/paper/2206.12191/) (2022)
- [Machine Learning in Short-Reach Optical Systems: A Comprehensive Survey](https://mlsanity.com/paper/2405.09557/) (2024)
- [Unconventional Computing based on Four Wave Mixing in Highly Nonlinear Waveguides](https://mlsanity.com/paper/2402.09135/) (2024)

## Abstract (arXiv)

We investigate the complexity and performance of recurrent neural network (RNN) models as post-processing units for the compensation of fibre nonlinearities in digital coherent systems carrying polarization multiplexed 16-QAM and 32-QAM signals. We evaluate three bi-directional RNN models, namely the bi-LSTM, bi-GRU and bi-Vanilla-RNN and show that all of them are promising nonlinearity compensators especially in dispersion unmanaged systems. Our simulations show that during inference the three models provide similar compensation performance, therefore in real-life systems the simplest scheme based on Vanilla-RNN units should be preferred. We compare bi-Vanilla-RNN with Volterra nonlinear equalizers and exhibit its superiority both in terms of performance and complexity, thus highlighting that RNN processing is a very promising pathway for the upgrade of long-haul optical communication systems utilizing coherent detection.