# Transformer-Based Decoding in Concatenated Coding Schemes Under Synchronization Errors

arXiv: https://arxiv.org/abs/2511.00999 · HTML: https://mlsanity.com/paper/2511.00999/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 2 Nov 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [Deep Learning Methods for Improved Decoding of Linear Codes](https://mlsanity.com/paper/1706.07043/) (2017)
- [Learning to Decode Linear Codes Using Deep Learning](https://mlsanity.com/paper/1607.04793/) (2016)
- [Error Correction Code Transformer](https://mlsanity.com/paper/2203.14966/) (2022)
- [Deepcode: Feedback Codes via Deep Learning](https://mlsanity.com/paper/1807.00801/) (2018)
- [Hyper-Graph-Network Decoders for Block Codes](https://mlsanity.com/paper/1909.09036/) (2019)
- [Deep Learning for Decoding of Linear Codes - A Syndrome-Based Approach](https://mlsanity.com/paper/1802.04741/) (2018)
- [Belief Propagation Neural Networks](https://mlsanity.com/paper/2007.00295/) (2020)
- [Neural Offset Min-Sum Decoding](https://mlsanity.com/paper/1701.05931/) (2017)
- [CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codes](https://mlsanity.com/paper/2405.01033/) (2024)
- [Deep Learning-based Polar Code Design](https://mlsanity.com/paper/1909.12035/) (2019)
- [Data-Driven Symbol Detection via Model-Based Machine Learning](https://mlsanity.com/paper/2002.07806/) (2020)
- [Data-Driven Factor Graphs for Deep Symbol Detection](https://mlsanity.com/paper/2002.00758/) (2020)
- [CRISP: Curriculum based Sequential Neural Decoders for Polar Code Family](https://mlsanity.com/paper/2210.00313/) (2022)
- [Deep DNA Storage: Scalable and Robust DNA Storage via Coding Theory and Deep Learning](https://mlsanity.com/paper/2109.00031/) (2021)
- [Deep Learning-Based Detection for Marker Codes over Insertion and Deletion Channels](https://mlsanity.com/paper/2401.01155/) (2024)

## Built on by (most cited first, 1 of 1)

- [MaskCode: Mask Transformer for Feedback-Assisted Coding With Linear Block Codes](https://mlsanity.com/paper/2609.00715/) (2026)

## Abstract (arXiv)

We consider the reconstruction of a codeword from multiple noisy copies that are independently corrupted by insertions, deletions, and substitutions. This problem arises, for example, in DNA data storage. A common code construction uses a concatenated coding scheme that combines an outer linear block code with an inner code, which can be either a nonlinear marker code or a convolutional code. Outer decoding is done with Belief Propagation, and inner decoding is done with the Bahl-Cocke-Jelinek-Raviv (BCJR) algorithm. However, the BCJR algorithm scales exponentially with the number of noisy copies, which makes it infeasible to reconstruct a codeword from more than about four copies. In this work, we introduce BCJRFormer, a transformer-based neural inner decoder. BCJRFormer achieves error rates comparable to the BCJR algorithm for binary and quaternary single-message transmissions of marker codes. Importantly, BCJRFormer scales quadratically with the number of noisy copies. This property makes BCJRFormer well-suited for DNA data storage, where multiple reads of the same DNA strand occur. To lower error rates, we replace the Belief Propagation outer decoder with a transformer-based decoder. Together, these modifications yield an efficient and performant end-to-end transformer-based pipeline for decoding multiple noisy copies affected by insertion, deletion, and substitution errors. Additionally, we propose a novel cross-attending transformer architecture called ConvBCJRFormer. This architecture extends BCJRFormer to decode transmissions of convolutional codewords, serving as an initial step toward joint inner and outer decoding for more general linear code classes.