# Geometric Inductive Biases for Semi-Supervised Equalization: The Constellation-Aware Transformer

arXiv: https://arxiv.org/abs/2609.33695 · HTML: https://mlsanity.com/paper/2609.33695/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [An Introduction to Deep Learning for the Physical Layer](https://mlsanity.com/paper/1702.00832/) (2017)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [Sionna: An Open-Source Library for Next-Generation Physical Layer Research](https://mlsanity.com/paper/2203.11854/) (2022)
- [Over the Air Deep Learning Based Radio Signal Classification](https://mlsanity.com/paper/1712.04578/) (2017)
- [Hierarchical Graph Network for Multi-hop Question Answering](https://mlsanity.com/paper/1911.03631/) (2019)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [DeepRx: Fully Convolutional Deep Learning Receiver](https://mlsanity.com/paper/2005.01494/) (2020)
- [Model-Driven Deep Learning for Physical Layer Communications](https://mlsanity.com/paper/1809.06059/) (2018)
- [Deep Learning for Decoding of Linear Codes - A Syndrome-Based Approach](https://mlsanity.com/paper/1802.04741/) (2018)
- [ViterbiNet: A Deep Learning Based Viterbi Algorithm for Symbol Detection](https://mlsanity.com/paper/1905.10750/) (2019)
- [DeepSIC: Deep Soft Interference Cancellation for Multiuser MIMO Detection](https://mlsanity.com/paper/2002.03214/) (2020)
- [Unsupervised Linear and Nonlinear Channel Equalization and Decoding using Variational Autoencoders](https://mlsanity.com/paper/1905.08795/) (2019)
- [Blind Channel Equalization using Variational Autoencoders](https://mlsanity.com/paper/1803.01526/) (2018)
- [Transformers are Provably Optimal In-context Estimators for Wireless Communications](https://mlsanity.com/paper/2311.00226/) (2023)
- [Learning How to Demodulate from Few Pilots via Meta-Learning](https://mlsanity.com/paper/1903.02184/) (2019)
- [Semi-Supervised Variational Inference over Nonlinear Channels](https://mlsanity.com/paper/2309.11841/) (2023)
- [Semi-supervised MIMO Detection Using Cycle-consistent Generative Adversarial Network](https://mlsanity.com/paper/2206.03686/) (2022)
- [Joint Channel Estimation and Feedback with Masked Token Transformers in Massive MIMO Systems](https://mlsanity.com/paper/2306.06125/) (2023)

## Abstract (arXiv)

Decoding signals over unknown channels with minimal pilot overhead is a critical challenge in next-generation communications. Existing deep learning approaches typically rely on generic encoders that struggle to model long-range temporal dependencies or efficiently capture the channel's physical properties from scarce data. We argue that standard architectures suffer from agnostic estimation gaps, as they must implicitly learn the constellation geometry that is already known. We introduce the Constellation-Aware Transformer (CAT), a novel architecture that explicitly injects geometric inductive biases into the equalization process. CAT is composed of a stack of custom TransFIRmer blocks, which use an "early interaction" paradigm to co-process received signals and ideal constellation symbols. Each block features a split Feed-Forward Network that applies a Finite Impulse Response (FIR)-inspired filter for deconvolution and a parallel MLP for geometric refinement. We show that this design is structurally aligned with the optimal linear (MIMO Wiener) receiver: its attention can implement a matched-filter bank, and its bidirectional FIR branch provides the non-causal filtering that block MMSE equalization requires. In the semi-supervised setting, CAT needs fewer pilots than VAE and standard Transformer baselines: on two of our three ISI channels, it reaches a lower SER with 64 pilots than they do with 128.