# Hybrid Mamba-Transformer Decoder for Error-Correcting Codes

arXiv: https://arxiv.org/abs/2505.17834 · HTML: https://mlsanity.com/paper/2505.17834/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [Learning to Decode Linear Codes Using Deep Learning](https://mlsanity.com/paper/1607.04793/) (2016)
- [Communication Algorithms via Deep Learning](https://mlsanity.com/paper/1805.09317/) (2018)
- [Deep Learning for Decoding of Linear Codes - A Syndrome-Based Approach](https://mlsanity.com/paper/1802.04741/) (2018)
- [Neural Offset Min-Sum Decoding](https://mlsanity.com/paper/1701.05931/) (2017)
- [CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes](https://mlsanity.com/paper/2405.01033/) (2024)
- [Autoregressive Belief Propagation for Decoding Block Codes](https://mlsanity.com/paper/2103.11780/) (2021)
- [Deep Quantum Error Correction](https://mlsanity.com/paper/2301.11930/) (2023)
- [Boosting Learning for LDPC Codes to Improve the Error-Floor Performance](https://mlsanity.com/paper/2310.07194/) (2023)
- [Learning Linear Block Error Correction Codes](https://mlsanity.com/paper/2405.04050/) (2024)
- [How to Mask in Error Correction Code Transformer: Systematic and Double Masking](https://mlsanity.com/paper/2308.08128/) (2023)

## Built on by (most cited first, 4 of 4)

- [Toward Uncertainty-Aware and Generalizable Neural Decoding for Quantum LDPC Codes](https://mlsanity.com/paper/2510.06257/) (2025)
- [Leveraging Code Automorphisms for Improved Syndrome-Based Neural Decoding](https://mlsanity.com/paper/2605.03620/) (2026)
- [Scalable Mamba-Based Message-Passing Neural Decoder for Error-Correcting Codes](https://mlsanity.com/paper/2605.10681/) (2026)
- [Score Based Error Correcting Code Decoder](https://mlsanity.com/paper/2605.28358/) (2026)

## Abstract (arXiv)

We introduce a novel deep learning method for decoding error correction codes based on the Mamba architecture, enhanced with Transformer layers. Our approach proposes a hybrid decoder that leverages Mamba's efficient sequential modeling while maintaining the global context capabilities of Transformers. To further improve performance, we design a novel layer-wise masking strategy applied to each Mamba layer, allowing selective attention to relevant code features at different depths. Additionally, we introduce a progressive layer-wise loss, supervising the network at intermediate stages and promoting robust feature extraction throughout the decoding process. Comprehensive experiments across a range of linear codes demonstrate that our method significantly outperforms Transformer-only decoders and standard Mamba models.