# Error Correction Code Transformer

arXiv: https://arxiv.org/abs/2203.14966 · HTML: https://mlsanity.com/paper/2203.14966/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 27 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [Sionna: An Open-Source Library for Next-Generation Physical Layer Research](https://mlsanity.com/paper/2203.11854/) (2022)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [Learning to Decode Linear Codes Using Deep Learning](https://mlsanity.com/paper/1607.04793/) (2016)
- [Communication Algorithms via Deep Learning](https://mlsanity.com/paper/1805.09317/) (2018)
- [Hyper-Graph-Network Decoders for Block Codes](https://mlsanity.com/paper/1909.09036/) (2019)
- [Deep Learning for Decoding of Linear Codes - A Syndrome-Based Approach](https://mlsanity.com/paper/1802.04741/) (2018)
- [Neural Offset Min-Sum Decoding](https://mlsanity.com/paper/1701.05931/) (2017)
- [Autoregressive Belief Propagation for Decoding Block Codes](https://mlsanity.com/paper/2103.11780/) (2021)
- [Geometric Transformer for End-to-End Molecule Properties Prediction](https://mlsanity.com/paper/2110.13721/) (2021)
- [perm2vec: Graph Permutation Selection for Decoding of Error Correction Codes using Self-Attention](https://mlsanity.com/paper/2002.02315/) (2020)

## Built on by (most cited first, 30 of 47)

- [Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimization](https://mlsanity.com/paper/2308.05384/) (2023)
- [SpectFormer: Frequency and Attention is what you need in a Vision Transformer](https://mlsanity.com/paper/2304.06446/) (2023)
- [Denoising Diffusion Error Correction Codes](https://mlsanity.com/paper/2209.13533/) (2022)
- [CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes](https://mlsanity.com/paper/2405.01033/) (2024)
- [Deep Quantum Error Correction](https://mlsanity.com/paper/2301.11930/) (2023)
- [Graph Neural Networks for Channel Decoding](https://mlsanity.com/paper/2207.14742/) (2022)
- [Boosting Learning for LDPC Codes to Improve the Error-Floor Performance](https://mlsanity.com/paper/2310.07194/) (2023)
- [Learning Linear Block Error Correction Codes](https://mlsanity.com/paper/2405.04050/) (2024)
- [CRISP: Curriculum based Sequential Neural Decoders for Polar Code Family](https://mlsanity.com/paper/2210.00313/) (2022)
- [How to Mask in Error Correction Code Transformer: Systematic and Double Masking](https://mlsanity.com/paper/2308.08128/) (2023)
- [Neural Decoding with Optimization of Node Activations](https://mlsanity.com/paper/2206.00786/) (2022)
- [Boosted Neural Decoders: Achieving Extreme Reliability of LDPC Codes for 6G Networks](https://mlsanity.com/paper/2405.13413/) (2024)
- [Factor Graph Optimization of Error-Correcting Codes for Belief Propagation Decoding](https://mlsanity.com/paper/2406.12900/) (2024)
- [Accelerating Error Correction Code Transformers](https://mlsanity.com/paper/2410.05911/) (2024)
- [Soft Graph Transformer for MIMO Detection](https://mlsanity.com/paper/2509.12694/) (2025)
- [Nested Construction of Polar Codes via Transformers](https://mlsanity.com/paper/2401.17188/) (2024)
- [On the Design and Performance of Machine Learning Based Error Correcting Decoders](https://mlsanity.com/paper/2410.15899/) (2024)
- [Toward Uncertainty-Aware and Generalizable Neural Decoding for Quantum LDPC Codes](https://mlsanity.com/paper/2510.06257/) (2025)
- [Generalization Bounds for Transformer Channel Decoders](https://mlsanity.com/paper/2601.06969/) (2026)
- [DoDo-Code: an Efficient Levenshtein Distance Embedding-based Code for 4-ary IDS Channel](https://mlsanity.com/paper/2312.12717/) (2023)
- [Deep Learning-Based Detection for Marker Codes over Insertion and Deletion Channels](https://mlsanity.com/paper/2401.01155/) (2024)
- [DeepPolar: Inventing Nonlinear Large-Kernel Polar Codes via Deep Learning](https://mlsanity.com/paper/2402.08864/) (2024)
- [Unified Error Correction Code Transformer with Low Complexity](https://mlsanity.com/paper/2410.03364/) (2024)
- [Cross-Attention Message-Passing Transformers for Code-Agnostic Decoding in 6G Networks](https://mlsanity.com/paper/2507.01038/) (2025)
- [Generative Feature Imputing -- A Technique for Error-resilient Semantic Communication](https://mlsanity.com/paper/2508.17957/) (2025)
- [Transformer-Based Decoding in Concatenated Coding Schemes Under Synchronization Errors](https://mlsanity.com/paper/2511.00999/) (2025)
- [Syndrome-Flow Consistency Model Achieves One-step Denoising Error Correction Codes](https://mlsanity.com/paper/2512.01389/) (2025)
- [In-Context Source and Channel Coding](https://mlsanity.com/paper/2601.10267/) (2026)
- [Adapting Diffusion Language Models for Lossless Pixel-Level Image Transmission](https://mlsanity.com/paper/2606.06273/) (2026)
- [Disturbance-based Discretization, Differentiable IDS Channel, and an IDS-Correcting Code for DNA-based Storage](https://mlsanity.com/paper/2407.18929/) (2024)

## Abstract (arXiv)

Error correction code is a major part of the communication physical layer, ensuring the reliable transfer of data over noisy channels. Recently, neural decoders were shown to outperform classical decoding techniques. However, the existing neural approaches present strong overfitting due to the exponential training complexity, or a restrictive inductive bias due to reliance on Belief Propagation. Recently, Transformers have become methods of choice in many applications thanks to their ability to represent complex interactions between elements. In this work, we propose to extend for the first time the Transformer architecture to the soft decoding of linear codes at arbitrary block lengths. We encode each channel's output dimension to high dimension for better representation of the bits information to be processed separately. The element-wise processing allows the analysis of the channel output reliability, while the algebraic code and the interaction between the bits are inserted into the model via an adapted masked self-attention module. The proposed approach demonstrates the extreme power and flexibility of Transformers and outperforms existing state-of-the-art neural decoders by large margins at a fraction of their time complexity.