# Deep-learning Autoencoder for Coherent and Nonlinear Optical Communication

arXiv: https://arxiv.org/abs/2006.15027 · HTML: https://mlsanity.com/paper/2006.15027/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2020.

## Built on by (most cited first, 4 of 4)

- [End-to-End Deep Learning of Long-Haul Coherent Optical Fiber Communications via Regular Perturbation Model](https://mlsanity.com/paper/2107.12320/) (2021)
- [End-to-End Learning of Joint Geometric and Probabilistic Constellation Shaping](https://mlsanity.com/paper/2112.05050/) (2021)
- [Model-Based Deep Learning of Joint Probabilistic and Geometric Shaping for Optical Communication](https://mlsanity.com/paper/2204.07457/) (2022)
- [Short-Length Code Designs for Integrated Sensing and Communications: A Deep Learning Approach](https://mlsanity.com/paper/2609.33605/) (2026)

## Abstract (arXiv)

Motivated by the recent success of end-to-end training of communications in the wireless domain, we strive to adapt the end-to-end-learning idea from the wireless case (i.e., linear) to coherent optical fiber links (i.e., nonlinear). Although, at first glance, it sounds like a straightforward extension, it turns out that several pitfalls exist - in terms of theory but also in terms of practical implementation. This paper analyzes the potential of an autoencoder and limitations for the optical fiber under the influence of Kerr-nonlinearity and chromatic dispersion. As there is no exact capacity limit known and, hence, no analytical perfect system solution available, we set great value to the interpretability on the learnings of the autoencoder. Therefore, we design its architecture to be as close as possible to the structure of a classic communication system, knowing that this may limit its degree of freedom and, thus, its performance. Nevertheless, we were able to achieve an unexpected high gain in terms of spectral efficiency compared to a conventional reference system.