# Pre-Whitening and BCJR Posterior Distillation for Bi-LSTM Detection in Faster-than-Nyquist Signaling

arXiv: https://arxiv.org/abs/2609.07762 · HTML: https://mlsanity.com/paper/2609.07762/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Builds on (most cited first, 10 of 10)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ViterbiNet: A Deep Learning Based Viterbi Algorithm for Symbol Detection](https://mlsanity.com/paper/1905.10750/) (2019)
- [Data-Driven Factor Graphs for Deep Symbol Detection](https://mlsanity.com/paper/2002.00758/) (2020)
- [Knowledge Distillation Applied to Optical Channel Equalization: Solving the Parallelization Problem of Recurrent Connection](https://mlsanity.com/paper/2212.04569/) (2022)
- [A Novel Domain-Aware CNN Architecture for Faster-than-Nyquist Signaling Detection](https://mlsanity.com/paper/2507.15291/) (2025)
- [A Novel CNN Based Standalone Detector for Faster-than-Nyquist Signaling](https://mlsanity.com/paper/2508.12964/) (2025)
- [Faster-Than-Nyquist Equalization with Convolutional Neural Networks](https://mlsanity.com/paper/2501.11594/) (2025)
- [Self-Attention Transformer-Based Detector for Faster-than-Nyquist Signaling](https://mlsanity.com/paper/2607.20745/) (2026)
- [A Comparative Analysis of MLP and Kolmogorov-Arnold Networks (KAN) for Faster-than-Nyquist (FTN) Signaling Detection](https://mlsanity.com/paper/2608.02062/) (2026)
- [Low-Complexity Recurrent Neural Network Detector for Faster-than-Nyquist Signaling](https://mlsanity.com/paper/2608.04155/) (2026)

## Abstract (arXiv)

Recurrent detectors such as bidirectional long short-term memory (Bi-LSTM) networks are low-complexity alternatives to the optimal Bahl-Cocke-Jelinek-Raviv (BCJR) detector for faster-than-Nyquist (FTN) signaling. Motivated by convolutional detectors that build the intersymbol interference (ISI) structure into their architecture, we ask whether processing nested ISI windows in separate recurrent branches improves the bit error rate (BER) of a Bi-LSTM. Across roughly 260 controlled trainings it does not: at a matched parameter budget and a matched readout, the multi-window architecture never significantly beats a plain Bi-LSTM. Nested windowing is an invertible rearrangement that adds no information, extra branches only add bottlenecks, and a distillation diagnostic shows the network is already near optimal for its window. The limitation is therefore the observation model, not the architecture. Keeping the architecture fixed, we pre-whiten the input, restoring the conditional independence that colored matched-filter noise violates, and distill the BCJR soft posterior into the network. With 3.4% more parameters this reaches 1.05 times the BCJR BER at a compression factor of 0.8 and 1.89 times at 0.7, improving to 1.47 times when the whitened window is widened. The 23.7% BER reduction at 0.8 requires an ill-conditioned ISI matrix but is not monotone in the conditioning, and it holds across five independent noise realizations and a symbol-level McNemar test.