# Gradient-free training of autoencoders for non-differentiable communication channels

arXiv: https://arxiv.org/abs/2012.11227 · HTML: https://mlsanity.com/paper/2012.11227/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 3 of 3)

- [Geometric Constellation Shaping for Fiber-Optic Channels via End-to-End Learning](https://mlsanity.com/paper/2211.04311/) (2022)
- [Deep Deterministic Policy Gradient for End-to-End Communication Systems without Prior Channel Knowledge](https://mlsanity.com/paper/2305.07448/) (2023)
- [Experimental End-to-End Optimization of Directly Modulated Laser-based IM/DD Transmission](https://mlsanity.com/paper/2508.19910/) (2025)

## Abstract (arXiv)

Training of autoencoders using the back-propagation algorithm is challenging for non-differential channel models or in an experimental environment where gradients cannot be computed. In this paper, we study a gradient-free training method based on the cubature Kalman filter. To numerically validate the method, the autoencoder is employed to perform geometric constellation shaping on differentiable communication channels, showing the same performance as the back-propagation algorithm. Further investigation is done on a non-differentiable communication channel that includes: laser phase noise, additive white Gaussian noise and blind phase search-based phase noise compensation. Our results indicate that the autoencoder can be successfully optimized using the proposed training method to achieve better robustness to residual phase noise with respect to standard constellation schemes such as Quadrature Amplitude Modulation and Iterative Polar Modulation for the considered conditions.