# Blind Equalization and Channel Estimation in Coherent Optical Communications Using Variational Autoencoders

arXiv: https://arxiv.org/abs/2204.11776 · HTML: https://mlsanity.com/paper/2204.11776/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 38 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Unsupervised Linear and Nonlinear Channel Equalization and Decoding using Variational Autoencoders](https://mlsanity.com/paper/1905.08795/) (2019)
- [Blind Channel Equalization using Variational Autoencoders](https://mlsanity.com/paper/1803.01526/) (2018)
- [Design of Communication Systems using Deep Learning: A Variational Inference Perspective](https://mlsanity.com/paper/1904.08559/) (2019)
- [Geometric Constellation Shaping for Phase-noise Channels Using a Differentiable Blind Phase Search](https://mlsanity.com/paper/2112.03114/) (2021)
- [Interplay of Probabilistic Shaping and the Blind Phase Search Algorithm](https://mlsanity.com/paper/1803.05957/) (2018)

## Built on by (most cited first, 7 of 7)

- [Blind Channel Estimation and Joint Symbol Detection with Data-Driven Factor Graphs](https://mlsanity.com/paper/2401.12627/) (2024)
- [Spiking Neural Network Decision Feedback Equalization](https://mlsanity.com/paper/2211.04756/) (2022)
- [Improving the Bootstrap of Blind Equalizers with Variational Autoencoders](https://mlsanity.com/paper/2301.06576/) (2023)
- [Optimization of Iterative Blind Detection based on Expectation Maximization and Belief Propagation](https://mlsanity.com/paper/2408.02312/) (2024)
- [Recent Advances on Machine Learning-aided DSP for Short-reach and Long-haul Optical Communications](https://mlsanity.com/paper/2411.10101/) (2024)
- [Novel Phase-Noise-Tolerant Variational-Autoencoder-Based Equalization Suitable for Space-Division-Multiplexed Transmission](https://mlsanity.com/paper/2509.14072/) (2025)
- [A Tensor-Train Framework for Bayesian Inference in High-Dimensional Systems: Applications to MIMO Detection and Channel Decoding](https://mlsanity.com/paper/2604.05890/) (2026)

## Abstract (arXiv)

We investigate the potential of adaptive blind equalizers based on variational inference for carrier recovery in optical communications. These equalizers are based on a low-complexity approximation of maximum likelihood channel estimation. We generalize the concept of variational autoencoder (VAE) equalizers to higher order modulation formats encompassing probabilistic constellation shaping (PCS), ubiquitous in optical communications, oversampling at the receiver, and dual-polarization transmission. Besides black-box equalizers based on convolutional neural networks, we propose a model-based equalizer based on a linear butterfly filter and train the filter coefficients using the variational inference paradigm. As a byproduct, the VAE also provides a reliable channel estimation. We analyze the VAE in terms of performance and flexibility over a classical additive white Gaussian noise (AWGN) channel with inter-symbol interference (ISI) and over a dispersive linear optical dual-polarization channel. We show that it can extend the application range of blind adaptive equalizers by outperforming the state-of-the-art constant-modulus algorithm (CMA) for PCS for both fixed but also time-varying channels. The evaluation is accompanied with a hyperparameter analysis.