# How to Mask in Error Correction Code Transformer: Systematic and Double Masking

arXiv: https://arxiv.org/abs/2308.08128 · HTML: https://mlsanity.com/paper/2308.08128/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deep Learning-Based Communication Over the Air](https://mlsanity.com/paper/1707.03384/) (2017)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [Learning to Decode Linear Codes Using Deep Learning](https://mlsanity.com/paper/1607.04793/) (2016)
- [Error Correction Code Transformer](https://mlsanity.com/paper/2203.14966/) (2022)
- [Communication Algorithms via Deep Learning](https://mlsanity.com/paper/1805.09317/) (2018)
- [Hyper-Graph-Network Decoders for Block Codes](https://mlsanity.com/paper/1909.09036/) (2019)
- [Deep Learning for Decoding of Linear Codes - A Syndrome-Based Approach](https://mlsanity.com/paper/1802.04741/) (2018)
- [Neural Offset Min-Sum Decoding](https://mlsanity.com/paper/1701.05931/) (2017)
- [Autoregressive Belief Propagation for Decoding Block Codes](https://mlsanity.com/paper/2103.11780/) (2021)

## Built on by (most cited first, 5 of 5)

- [CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes](https://mlsanity.com/paper/2405.01033/) (2024)
- [Learning Linear Block Error Correction Codes](https://mlsanity.com/paper/2405.04050/) (2024)
- [Hybrid Mamba-Transformer Decoder for Error-Correcting Codes](https://mlsanity.com/paper/2505.17834/) (2025)
- [Accelerating Error Correction Code Transformers](https://mlsanity.com/paper/2410.05911/) (2024)
- [Score Based Error Correcting Code Decoder](https://mlsanity.com/paper/2605.28358/) (2026)

## Abstract (arXiv)

In communication and storage systems, error correction codes (ECCs) are pivotal in ensuring data reliability. As deep learning's applicability has broadened across diverse domains, there is a growing research focus on neural network-based decoders that outperform traditional decoding algorithms. Among these neural decoders, Error Correction Code Transformer (ECCT) has achieved the state-of-the-art performance, outperforming other methods by large margins. To further enhance the performance of ECCT, we propose two novel methods. First, leveraging the systematic encoding technique of ECCs, we introduce a new masking matrix for ECCT, aiming to improve the performance and reduce the computational complexity. Second, we propose a novel transformer architecture of ECCT called a double-masked ECCT. This architecture employs two different mask matrices in a parallel manner to learn more diverse features of the relationship between codeword bits in the masked self-attention blocks. Extensive simulation results show that the proposed double-masked ECCT outperforms the conventional ECCT, achieving the state-of-the-art decoding performance with significant margins.