# CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes

arXiv: https://arxiv.org/abs/2405.01033 · HTML: https://mlsanity.com/paper/2405.01033/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [Learning to Decode Linear Codes Using Deep Learning](https://mlsanity.com/paper/1607.04793/) (2016)
- [Error Correction Code Transformer](https://mlsanity.com/paper/2203.14966/) (2022)
- [Communication Algorithms via Deep Learning](https://mlsanity.com/paper/1805.09317/) (2018)
- [Hyper-Graph-Network Decoders for Block Codes](https://mlsanity.com/paper/1909.09036/) (2019)
- [Deep Learning for Decoding of Linear Codes - A Syndrome-Based Approach](https://mlsanity.com/paper/1802.04741/) (2018)
- [Neural Offset Min-Sum Decoding](https://mlsanity.com/paper/1701.05931/) (2017)
- [Denoising Diffusion Error Correction Codes](https://mlsanity.com/paper/2209.13533/) (2022)
- [Autoregressive Belief Propagation for Decoding Block Codes](https://mlsanity.com/paper/2103.11780/) (2021)
- [Boosting Learning for LDPC Codes to Improve the Error-Floor Performance](https://mlsanity.com/paper/2310.07194/) (2023)
- [How to Mask in Error Correction Code Transformer: Systematic and Double Masking](https://mlsanity.com/paper/2308.08128/) (2023)

## Built on by (most cited first, 21 of 21)

- [Hybrid Mamba-Transformer Decoder for Error-Correcting Codes](https://mlsanity.com/paper/2505.17834/) (2025)
- [Accelerating Error Correction Code Transformers](https://mlsanity.com/paper/2410.05911/) (2024)
- [Soft Graph Transformer for MIMO Detection](https://mlsanity.com/paper/2509.12694/) (2025)
- [On the Design and Performance of Machine Learning Based Error Correcting Decoders](https://mlsanity.com/paper/2410.15899/) (2024)
- [Generalization Bounds for Transformer Channel Decoders](https://mlsanity.com/paper/2601.06969/) (2026)
- [Cross-Attention Message-Passing Transformers for Code-Agnostic Decoding in 6G Networks](https://mlsanity.com/paper/2507.01038/) (2025)
- [Transformer-Based Decoding in Concatenated Coding Schemes Under Synchronization Errors](https://mlsanity.com/paper/2511.00999/) (2025)
- [Syndrome-Flow Consistency Model Achieves One-step Denoising Error Correction Codes](https://mlsanity.com/paper/2512.01389/) (2025)
- [Disturbance-based Discretization, Differentiable IDS Channel, and an IDS-Correcting Code for DNA-based Storage](https://mlsanity.com/paper/2407.18929/) (2024)
- [VT-Former: Efffcient Transformer-based Decoder for Varshamov-Tenengolts Codes](https://mlsanity.com/paper/2502.21060/) (2025)
- [On the Optimality of Single-label and Multi-label Neural Network Decoders](https://mlsanity.com/paper/2503.18758/) (2025)
- [Qubit-centric Transformer for Surface Code Decoding](https://mlsanity.com/paper/2510.11593/) (2025)
- [SAQ: Stabilizer-Aware Quantum Error Correction Decoder](https://mlsanity.com/paper/2512.08914/) (2025)
- [On the Fragility of AI-Based Channel Decoders under Small Channel Perturbations](https://mlsanity.com/paper/2602.01582/) (2026)
- [Soft Graph Diffusion Transformer for MIMO Detection](https://mlsanity.com/paper/2605.00449/) (2026)
- [Leveraging Code Automorphisms for Improved Syndrome-Based Neural Decoding](https://mlsanity.com/paper/2605.03620/) (2026)
- [Scalable Mamba-Based Message-Passing Neural Decoder for Error-Correcting Codes](https://mlsanity.com/paper/2605.10681/) (2026)
- [Score Based Error Correcting Code Decoder](https://mlsanity.com/paper/2605.28358/) (2026)
- [MaskCode: Mask Transformer for Feedback-Assisted Coding With Linear Block Codes](https://mlsanity.com/paper/2609.00715/) (2026)
- [ZeroCode: On-demand Error-Correcting Code Construction from the Zero Matrix via Reinforcement Learning](https://mlsanity.com/paper/2609.34265/) (2026)
- [Distance Prediction for Quantum Codes](https://mlsanity.com/paper/2610.18130/) (2026)

## Abstract (arXiv)

Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders. While conventional transformer-based decoders employ self-attention mechanism without distinguishing between the types of input vectors (i.e., magnitude and syndrome vectors), CrossMPT updates the two types of input vectors separately and iteratively using two masked cross-attention blocks. The mask matrices are determined by the code's parity-check matrix, which explicitly captures the irrelevant relationship between two input vectors. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement, while significantly reducing the memory usage, complexity, inference time, and training time.