Accelerating Error Correction Code Transformers
Not on the map. Why?
Abstract (arXiv)
Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with multiplication-free linear layers. We present an optimized self-attention mechanism to reduce computational complexity via codeaware multi-heads processing. Finally, we provide positional encoding via the Tanner graph eigendecomposition, enabling a richer representation of the graph connectivity. The approach not only matches or surpasses ECCT's performance but also significantly reduces energy consumption, memory footprint, and computational complexity. Our method brings transformer-based error correction closer to practical implementation in resource-constrained environments, achieving a 90% compression ratio and reducing arithmetic operation energy consumption by at least 224 times on modern hardware.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 8 Oct 2024.
Builds on
39 of 50 references.
References · 39
- 2017Attention Is All You Needcited by 59,932
- 2016Gaussian Error Linear Units (GELUs)cited by 3,431
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2020Reformer: The Efficient Transformercited by 1,658
- 2020Rethinking Attention with Performerscited by 1,584
- 2019Generating Long Sequences with Sparse Transformerscited by 1,523
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
- 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
27 more
- 2020A Generalization of Transformer Networks to Graphscited by 653
- 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
- 2022Efficiently Scaling Transformer Inferencecited by 529
- 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 471
- 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2021Graph Neural Networks with Learnable Structural and Positional Representationscited by 326
- 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
- 2023A Survey on Model Compression for Large Language Modelscited by 263
- 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
- 2023Efficient Large Language Models: A Surveycited by 183
- 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 176
- 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
- 2020Directional Graph Networkscited by 132
- 2017Deep Learning Methods for Improved Decoding of Linear Codescited by 66
- 2016Learning to Decode Linear Codes Using Deep Learningcited by 60
- 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
- 2022Error Correction Code Transformercited by 47
- 2018Communication Algorithms via Deep Learningcited by 42
- 2019Hyper-Graph-Network Decoders for Block Codescited by 42
- 2018Deep Learning for Decoding of Linear Codes - A Syndrome-Based Approachcited by 34
- 2022Denoising Diffusion Error Correction Codescited by 23
- 2024CrossMPT: Cross-attention Message-Passing Transformer for Error Correcting Codescited by 20
- 2024L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Modelscited by 15
- 2023Deep Quantum Error Correctioncited by 10
- 2024Learning Linear Block Error Correction Codescited by 8
- 2023How to Mask in Error Correction Code Transformer: Systematic and Double Maskingcited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.