# Blind Channel Equalization Using Vector-Quantized Variational Autoencoders

arXiv: https://arxiv.org/abs/2302.11687 · HTML: https://mlsanity.com/paper/2302.11687/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2023.

## Built on by (most cited first, 2 of 2)

- [At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence](https://mlsanity.com/paper/2402.18587/) (2024)
- [Recent Advances on Machine Learning-aided DSP for Short-reach and Long-haul Optical Communications](https://mlsanity.com/paper/2411.10101/) (2024)

## Abstract (arXiv)

State-of-the-art high-spectral-efficiency communication systems employ high-order modulation formats coupled with high symbol rates to accommodate the ever-growing demand for data rate-hungry applications. However, such systems are more vulnerable to linear and nonlinear transmission impairments, and it is important to mitigate the performance loss via digital signal processing. In this paper, we propose a novel machine learning approach for blind channel equalization and estimation using the vector quantized (VQ) acVAE framework. The proposed approach generalizes the applicability of the conventional acVAE-based equalizer to nonlinear systems employing high-order modulation formats by introducing a codebook component and an associated novel loss function. We evaluate the performance of the proposed method over a linear additive white Gaussian noise channel with intersymbol interference and two nonlinear scenarios. Simulation results show that the proposed method can achieve similar performance as a data aided equalizer using the acfMMSE criterion, and outperforms the blindacCMA and the acVAE-based channel equalizer. Furthermore, we show that for the linear channel, the proposed scheme exhibits better convergence properties than the acMMSE-based, the acCMA-based, and the acVAE-based equalizers in terms of both convergence speed and robustness to variations in training batch size and learning rate.