Neural Network Architectures for Optical Channel Nonlinear Compensation in Digital Subcarrier Multiplexing Systems

In this work, we propose to use various artificial neural network (ANN) structures for modeling and compensation of intra- and inter-subcarrier fiber nonlinear interference in digital subcarrier multiplexing (DSCM) optical transmission systems. We perform nonlinear channel equalization by employing different ANN cores including convolutional neural networks (CNN) and long short-term memory (LSTM) layers. We start to compensate the fiber nonlinearity distortion in DSCM systems by a fully connected network across all subcarriers. In subsequent steps, and borrowing from fiber nonlinearity analysis, we gradually upgrade the designs towards modular structures with better performance-complexity advantages. Our study shows that putting proper macro structures in design of ANN nonlinear equalizers in DSCM systems can be crucial for practical solutions in future generations of coherent optical transceivers.