# AttentionCode: Ultra-Reliable Feedback Codes for Short-Packet Communications

arXiv: https://arxiv.org/abs/2205.14955 · HTML: https://mlsanity.com/paper/2205.14955/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 52 references that are in the map.
- First posted on arXiv 30 May 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [DeepJSCC-f: Deep Joint Source-Channel Coding of Images with Feedback](https://mlsanity.com/paper/1911.11174/) (2019)
- [Deepcode: Feedback Codes via Deep Learning](https://mlsanity.com/paper/1807.00801/) (2018)
- [Federated Edge Learning with Misaligned Over-The-Air Computation](https://mlsanity.com/paper/2102.13604/) (2021)
- [Transformer-Empowered 6G Intelligent Networks: From Massive MIMO Processing to Semantic Communication](https://mlsanity.com/paper/2205.03770/) (2022)
- [Deep Extended Feedback Codes](https://mlsanity.com/paper/2105.01365/) (2021)
- [DRF Codes: Deep SNR-Robust Feedback Codes](https://mlsanity.com/paper/2112.11789/) (2021)
- [Simple Modulo can Significantly Outperform Deep Learning-based Deepcode](https://mlsanity.com/paper/2008.01686/) (2020)
- [Deepcode and Modulo-SK are Designed for Different Settings](https://mlsanity.com/paper/2008.07997/) (2020)

## Built on by (most cited first, 8 of 8)

- [Distributed Deep Joint Source-Channel Coding over a Multiple Access Channel](https://mlsanity.com/paper/2211.09920/) (2022)
- [All you need is feedback: Communication with block attention feedback codes](https://mlsanity.com/paper/2206.09457/) (2022)
- [Semantic Communications with Discrete-time Analog Transmission: A PAPR Perspective](https://mlsanity.com/paper/2208.08342/) (2022)
- [Feedback is Good, Active Feedback is Better: Block Attention Active Feedback Codes](https://mlsanity.com/paper/2211.01730/) (2022)
- [LightCode: Light Analytical and Neural Codes for Channels with Feedback](https://mlsanity.com/paper/2403.10751/) (2024)
- [Learning to Interfere in Non-Orthogonal Multiple-Access Joint Source-Channel Coding](https://mlsanity.com/paper/2504.03690/) (2025)
- [Deep Variable-Length Feedback Codes](https://mlsanity.com/paper/2602.07881/) (2026)
- [MaskCode: Mask Transformer for Feedback-Assisted Coding With Linear Block Codes](https://mlsanity.com/paper/2609.00715/) (2026)

## Abstract (arXiv)

Ultra-reliable short-packet communication is a major challenge in future wireless networks with critical applications. To achieve ultra-reliable communications beyond 99.999%, this paper envisions a new interaction-based communication paradigm that exploits feedback from the receiver. We present AttentionCode, a new class of feedback codes leveraging deep learning (DL) technologies. The underpinnings of AttentionCode are three architectural innovations: AttentionNet, input restructuring, and adaptation to fading channels, accompanied by several training methods, including large-batch training, distributed learning, look-ahead optimizer, training-test signal-to-noise ratio (SNR) mismatch, and curriculum learning. The training methods can potentially be generalized to other wireless communication applications with machine learning. Numerical experiments verify that AttentionCode establishes a new state of the art among all DL-based feedback codes in both additive white Gaussian noise (AWGN) channels and fading channels. In AWGN channels with noiseless feedback, for example, AttentionCode achieves a block error rate (BLER) of 10-7 when the forward channel SNR is 0 dB for a block size of 50 bits, demonstrating the potential of AttentionCode to provide ultra-reliable short-packet communications.