# Geometric Constellation Shaping for Fiber-Optic Channels via End-to-End Learning

arXiv: https://arxiv.org/abs/2211.04311 · HTML: https://mlsanity.com/paper/2211.04311/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 52 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017)
- [Model-free Training of End-to-end Communication Systems](https://mlsanity.com/paper/1812.05929/) (2018)
- [Approximating the Void: Learning Stochastic Channel Models from Observation with Variational Generative Adversarial Networks](https://mlsanity.com/paper/1805.06350/) (2018)
- [End-to-End Learning of Communications Systems Without a Channel Model](https://mlsanity.com/paper/1804.02276/) (2018)
- [End-to-end Deep Learning of Optical Fiber Communications](https://mlsanity.com/paper/1804.04097/) (2018)
- [Achievable Information Rates for Fiber Optics: Applications and Computations](https://mlsanity.com/paper/1709.10393/) (2017)
- [Concept and Experimental Demonstration of Optical IM/DD End-to-End System Optimization using a Generative Model](https://mlsanity.com/paper/1912.05146/) (2019)
- [Achievable Information Rates for Nonlinear Fiber Communication via End-to-end Autoencoder Learning](https://mlsanity.com/paper/1804.07675/) (2018)
- [End-to-End Optimized Transmission over Dispersive Intensity-Modulated Channels Using Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1901.08570/) (2019)
- [End-to-end Learning for GMI Optimized Geometric Constellation Shape](https://mlsanity.com/paper/1907.08535/) (2019)
- [End-to-End Learning of Geometrical Shaping Maximizing Generalized Mutual Information](https://mlsanity.com/paper/1912.05638/) (2019)
- [End-to-end optimization of coherent optical communications over the split-step Fourier method guided by the nonlinear Fourier transform theory](https://mlsanity.com/paper/2009.05324/) (2020)
- [End-to-end Autoencoder for Superchannel Transceivers with Hardware Impairment](https://mlsanity.com/paper/2103.15856/) (2021)
- [Model-Based End-to-End Learning for WDM Systems With Transceiver Hardware Impairments](https://mlsanity.com/paper/2111.14515/) (2021)
- [Deep Learning of Geometric Constellation Shaping including Fiber Nonlinearities](https://mlsanity.com/paper/1805.03785/) (2018)
- [Gradient-free training of autoencoders for non-differentiable communication channels](https://mlsanity.com/paper/2012.11227/) (2020)
- [Geometric Constellation Shaping for Phase-noise Channels Using a Differentiable Blind Phase Search](https://mlsanity.com/paper/2112.03114/) (2021)
- [Interplay of Probabilistic Shaping and the Blind Phase Search Algorithm](https://mlsanity.com/paper/1803.05957/) (2018)
- [End-to-End Deep Learning of Long-Haul Coherent Optical Fiber Communications via Regular Perturbation Model](https://mlsanity.com/paper/2107.12320/) (2021)
- [End-to-end Learning of a Constellation Shape Robust to Channel Condition Uncertainties](https://mlsanity.com/paper/2111.08302/) (2021)
- [End-to-End Learning of Joint Geometric and Probabilistic Constellation Shaping](https://mlsanity.com/paper/2112.05050/) (2021)
- [Model-Based Deep Learning of Joint Probabilistic and Geometric Shaping for Optical Communication](https://mlsanity.com/paper/2204.07457/) (2022)
- [Geometric Constellation Shaping for Fiber Optic Communication Systems via End-to-end Learning](https://mlsanity.com/paper/1810.00774/) (2018)
- [End-to-end Learning of a Constellation Shape Robust to Variations in SNR and Laser Linewidth](https://mlsanity.com/paper/2106.00431/) (2021)
- [Over-the-fiber Digital Predistortion Using Reinforcement Learning](https://mlsanity.com/paper/2106.04934/) (2021)
- [Optimization of Geometric Constellation Shaping for Wiener Phase Noise Channels with Varying Channel Parameters](https://mlsanity.com/paper/2206.12288/) (2022)

## Built on by (most cited first, 2 of 2)

- [Experimental End-to-End Optimization of Directly Modulated Laser-based IM/DD Transmission](https://mlsanity.com/paper/2508.19910/) (2025)
- [End-to-End Optical Semantic Communication over a Nonlinear WDM Fiber Link](https://mlsanity.com/paper/2609.37551/) (2026)

## Abstract (arXiv)

End-to-end learning has become a popular method to optimize a constellation shape of a communication system. When the channel model is differentiable, end-to-end learning can be applied with conventional backpropagation algorithm for optimization of the shape. A variety of optimization algorithms have also been developed for end-to-end learning over a non-differentiable channel model. In this paper, we compare gradient-free optimization method based on the cubature Kalman filter, model-free optimization and backpropagation for end-to-end learning on a fiber-optic channel modeled by the split-step Fourier method. The results indicate that the gradient-free optimization algorithms provide a decent replacement to backpropagation in terms of performance at the expense of computational complexity. Furthermore, the quantization problem of finite bit resolution of the digital-to-analog and analog-to-digital converters is addressed and its impact on geometrically shaped constellations is analysed. Here, the results show that when optimizing a constellation with respect to mutual information, a minimum number of quantization levels is required to achieve shaping gain. For generalized mutual information, the gain is maintained throughout all of the considered quantization levels. Also, the results implied that the autoencoder can adapt the constellation size to the given channel conditions.